pictuga
/
morss
1
1
Fork 0
Code Issues Pull Requests Actions Packages Releases Activity
222 Commits
1 Branch
2 Tags
3.5 MiB
Commit Graph

151 Commits (eafbc14a587b3f6ea58b43620d14006640faf1b7)

Author SHA1 Message Date
pictuga eafbc14a58 Allow "progress" in JSON 2013-12-21 20:09:14 +01:00
pictuga cbdb61a918 Allow real python dict in Cache 
Makes it possible to store things a nicer way
 2013-12-21 17:05:12 +01:00
pictuga 2a8c94fd88 Work with unicode strings 
In downloads at least (I don't quite remember what readability returns actually, but "it just works")
 2013-12-12 23:12:33 +01:00
pictuga eac2e7a79a Try to avoid erros with saving 
Mainly: do not create empty cache file, and accept empty cache file (doesn't sound logical but still)
 2013-12-12 16:41:58 +01:00
pictuga d2345a94f8 Speed up a little html encoding detection 
Not sure this is a smart move
 2013-12-07 20:35:32 +01:00
pictuga 245ba99ae9 Always clean up downloads' enconding 
TPB, fuck ye
 2013-12-07 20:33:33 +01:00
pictuga e2512dba0c Cath all errors in threads 
Makes in possible to empty the queue whatsoever happens
 2013-12-01 19:45:00 +01:00
pictuga 1644a409ef Add application/xml as html mimetype 2013-12-01 15:48:17 +01:00
pictuga 8b3062eaa2 Make Accept headers py 2.6 compatible 2013-12-01 15:47:20 +01:00
pictuga 66c6dcf203 Store "as it" in JSON (int or str) 
The json lib does all the work
 2013-12-01 15:46:02 +01:00
pictuga 52fe67beb3 Create cache file when missing 
Damn it...
 2013-12-01 15:45:21 +01:00
pictuga 020c095194 Pretty-print JSON to cache 
Easier for debugging, not really needed though, takes more place for nothing
 2013-12-01 15:44:19 +01:00
pictuga c83f5bb23d URI based Cache filename 
Problem: lengthy filenames, have to cut them after 255 chars, sucks with FB, can cause problems
 2013-12-01 15:42:35 +01:00
pictuga 1a50cb390a User JSON to save cache 
Faster, cleaner, safer, unobfuscated, not 1/3 bigger
 2013-11-30 19:59:19 +01:00
pictuga fde5d3e8c5 Argument "silent" prevents from outputting feed 
Also fixes small indentation typo (non blocking though)
 2013-11-30 19:39:27 +01:00
pictuga 1d10d30173 Reddit: check feed url instead of link url 2013-11-30 19:36:44 +01:00
pictuga b927a4d089 Use "str.startswith" instead of ugly hack 2013-11-30 19:36:15 +01:00
pictuga 8916216403 Add "txt" argument, to change HTTP Content-Type 2013-11-30 19:35:24 +01:00
pictuga 336d98230f Make sure final data is there in downloader 2013-11-30 19:34:34 +01:00
pictuga 2634dab40c Use HTTP Accept headers 
Tries to tell servers what we need, hopefull this will avoid the useless donwloading of pdf, images and so on (especially on Hacker News)
 2013-11-30 19:33:36 +01:00
pictuga 0de5adc505 Remove return value in one __ini__ 
It was some kind of previous code left-over
 2013-11-30 19:30:54 +01:00
pictuga 363014ee4c User IOError to catch errors 
Most http related exceptions inherit from that one
 2013-11-30 17:33:57 +01:00
pictuga 7156dd5522 Support for non-textual gzipped http content 2013-11-24 21:55:07 +01:00
pictuga ecc18dc4ab Thread Gather() 
Twice faster. Ctrl-C no longer works in terminal. Don't know how to fix it. Using threading.active_count() or sth didn't work as expected. New settings (THREADS) to change the number of threads to use.
 2013-11-24 20:52:53 +01:00
pictuga 8832611a05 Fix MAX_ITEM condition 2013-11-24 20:49:14 +01:00
pictuga 1274df4e1d Yet another feedburner workaround 
This one really drove me mad. When the "Referer" is feedburner itself, it just will show html page (with Google Reader Add buttons), instead of the regular feed. Itself adding "?format=xml" at the end of the url also solves it, but users don't know about that...
 2013-11-24 20:43:09 +01:00
pictuga d3db04332e Don't run Fix() on dropped elements 2013-11-24 17:36:38 +01:00
pictuga f94bc4545e Add "LIM_TIME" to drop elements based on time 
Also this value and the similars, all take -1 as unlimited value
 2013-11-16 16:02:35 +01:00
pictuga 417609c1f0 Fix crash on empty html pages 
Was due to "chardet" returning "None", which then crashed "str.replace"
 2013-11-16 16:00:07 +01:00
pictuga 6975772ef9 Revert "Improve facebook and feedburner workarounds" 
This PARTIALLY reverts commit 8a5202efd5 (only the feedburner part).
 2013-11-15 16:54:41 +01:00
pictuga 3a784be93e End of typo fox in cachePath 2013-11-11 00:34:32 +01:00
pictuga ddc9e4e3cb Fix typo around cachePath 
Identation was wrong in if/else statement
 2013-11-10 23:55:44 +01:00
pictuga 88e7694de9 Add "proxy" mode: does't make the feed full-text 
Useful for feedify (twitter, facebook, etc) and for bad written rss feeds in not-strong-enough news readers (eg. TPB on tt-rss). Uses "persistent" in Cache() so as not to empty the cache because of that (cause Cache() drops items not asked for).
 2013-11-10 23:52:04 +01:00
pictuga 3664879d92 Add full FB API (Graph API) support 2013-11-09 18:48:06 +01:00
pictuga e5fc9786f6 Set urlPath var earlier 2013-11-09 18:43:16 +01:00
pictuga 4ece739722 "redirect" and "url" in cache to control end url 
Allow redirection, while keeping the same cache object
 2013-11-09 18:40:23 +01:00
pictuga 6c72a6e583 Added "persistent" mode in Cache 
Keep not-reused vars in cache
 2013-11-09 18:38:02 +01:00
pictuga f52e2871ac Allow get() on new values in Cache() 
Previously, it was only possible to access already-cached values via Cache.get(), this restriction is now gone. Also, data is now kept raw (not base64). It's also possible from now on to pass the type of var wanted (such as int).
 2013-11-03 20:18:43 +01:00
pictuga f756a4d834 Use more obvious varname for cache path 2013-11-03 13:32:24 +01:00
pictuga a0c75238b9 Add ability to redirect feeds 
Useful for websites with API (facebook). Will be in use later.
 2013-11-03 13:30:16 +01:00
pictuga 7504cfed2a Add 'theforce' option to redownload xml 
Useful for debugging
 2013-11-03 13:29:35 +01:00
pictuga 5737adc253 Improve Cache() 
Add the ability to create new Cache() objects from the current one (in the same directory). Allow the storage of int() (via str()). Don't store the key in the feed (useless, and not keeping it might make this service a little more less privacy-infrighting).
 2013-11-03 13:26:56 +01:00
pictuga 09ba8ca644 Use HOLD in log() 
So as not to write to the output when using on an HTTP server, during the writing of headers
 2013-11-03 13:24:27 +01:00
pictuga 97acdd9530 Custom "getopt" as Class 
Easier to use, an can now take value:
* CLI: morss key=value http://...
* HTTP: http://path/to/morss/:key=value/http://...
 2013-11-03 13:18:09 +01:00
pictuga 53c9b07d19 Split Fill() into Fix and Fill 
Fix improves links and suck things
 2013-10-22 20:55:24 +02:00
pictuga 71f20c8b29 Don't fill facebook for now 2013-10-21 21:31:02 +02:00
pictuga 8a5202efd5 Improve facebook and feedburner workarounds 2013-10-21 21:30:31 +02:00
pictuga 1f40f2a099 Add support for JSON APIs in feedify 2013-10-21 21:28:43 +02:00
pictuga 1802827d31 Fix decoding 
Was dropped in previous commit by mistake
 2013-10-21 21:24:19 +02:00
pictuga a1f9fe24c8 Use one single urllib2 handler 
Makes feedify on html pages more powerful
 2013-10-21 21:17:52 +02:00