129 Commits

Author SHA1 Message Date
7156dd5522 Support for non-textual gzipped http content 2013-11-24 21:55:07 +01:00
ecc18dc4ab Thread Gather()
Twice faster. Ctrl-C no longer works in terminal. Don't know how to fix it. Using threading.active_count() or sth didn't work as expected. New settings (THREADS) to change the number of threads to use.
2013-11-24 20:52:53 +01:00
8832611a05 Fix MAX_ITEM condition 2013-11-24 20:49:14 +01:00
1274df4e1d Yet another feedburner workaround
This one really drove me mad. When the "Referer" is feedburner itself, it just will show html page (with Google Reader Add buttons), instead of the regular feed. Itself adding "?format=xml" at the end of the url also solves it, but users don't know about that...
2013-11-24 20:43:09 +01:00
d3db04332e Don't run Fix() on dropped elements 2013-11-24 17:36:38 +01:00
f94bc4545e Add "LIM_TIME" to drop elements based on time
Also this value and the similars, all take -1 as unlimited value
2013-11-16 16:02:35 +01:00
417609c1f0 Fix crash on empty html pages
Was due to "chardet" returning "None", which then crashed "str.replace"
2013-11-16 16:00:07 +01:00
6975772ef9 Revert "Improve facebook and feedburner workarounds"
This PARTIALLY reverts commit 8a5202efd5cc9ea72c57a53ebb945ffab6eb32ea (only the feedburner part).
2013-11-15 16:54:41 +01:00
3a784be93e End of typo fox in cachePath 2013-11-11 00:34:32 +01:00
ddc9e4e3cb Fix typo around cachePath
Identation was wrong in if/else statement
2013-11-10 23:55:44 +01:00
88e7694de9 Add "proxy" mode: does't make the feed full-text
Useful for feedify (twitter, facebook, etc) and for bad written rss feeds in not-strong-enough news readers (eg. TPB on tt-rss). Uses "persistent" in Cache() so as not to empty the cache because of that (cause Cache() drops items not asked for).
2013-11-10 23:52:04 +01:00
3664879d92 Add full FB API (Graph API) support 2013-11-09 18:48:06 +01:00
e5fc9786f6 Set urlPath var earlier 2013-11-09 18:43:16 +01:00
4ece739722 "redirect" and "url" in cache to control end url
Allow redirection, while keeping the same cache object
2013-11-09 18:40:23 +01:00
6c72a6e583 Added "persistent" mode in Cache
Keep not-reused vars in cache
2013-11-09 18:38:02 +01:00
f52e2871ac Allow get() on new values in Cache()
Previously, it was only possible to access already-cached values via Cache.get(), this restriction is now gone. Also, data is now kept raw (not base64). It's also possible from now on to pass the type of var wanted (such as int).
2013-11-03 20:18:43 +01:00
f756a4d834 Use more obvious varname for cache path 2013-11-03 13:32:24 +01:00
a0c75238b9 Add ability to redirect feeds
Useful for websites with API (facebook). Will be in use later.
2013-11-03 13:30:16 +01:00
7504cfed2a Add 'theforce' option to redownload xml
Useful for debugging
2013-11-03 13:29:35 +01:00
5737adc253 Improve Cache()
Add the ability to create new Cache() objects from the current one (in the same directory). Allow the storage of int() (via str()). Don't store the key in the feed (useless, and not keeping it might make this service a little more less privacy-infrighting).
2013-11-03 13:26:56 +01:00
09ba8ca644 Use HOLD in log()
So as not to write to the output when using on an HTTP server, during the writing of headers
2013-11-03 13:24:27 +01:00
97acdd9530 Custom "getopt" as Class
Easier to use, an can now take value:
* CLI: morss key=value http://...
* HTTP: http://path/to/morss/:key=value/http://...
2013-11-03 13:18:09 +01:00
53c9b07d19 Split Fill() into Fix and Fill
Fix improves links and suck things
2013-10-22 20:55:24 +02:00
71f20c8b29 Don't fill facebook for now 2013-10-21 21:31:02 +02:00
8a5202efd5 Improve facebook and feedburner workarounds 2013-10-21 21:30:31 +02:00
1f40f2a099 Add support for JSON APIs in feedify 2013-10-21 21:28:43 +02:00
1802827d31 Fix decoding
Was dropped in previous commit by mistake
2013-10-21 21:24:19 +02:00
a1f9fe24c8 Use one single urllib2 handler
Makes feedify on html pages more powerful
2013-10-21 21:17:52 +02:00
76b81a31ec Drop desc by default
Arguably the reason is that users tend to believe the program does nothing, cause Firefox (and others?) only shows the description in the feed preview. The "keep" parameter keeps the description in place.
2013-10-02 12:06:14 +02:00
ae22104831 Center "* * *" in clip mode 2013-10-02 12:05:52 +02:00
145f59b546 Fix already-filled feeds
By pulling the content from "content" when needed
2013-10-01 19:50:28 +02:00
ae5e947417 Add support for facebook, google links 2013-10-01 19:49:53 +02:00
cf5040020e Use desktop browser UA for xml
Convenient for rss links detection and "feedify"
2013-10-01 19:47:45 +02:00
e3c1cd8619 Accept more types as "text" before readability 2013-10-01 19:47:06 +02:00
78706952fe Remove "clip" from Fill
Put that in Gather. Also removed from feeds.py. "alone" mode was also added (it removes the description).
2013-10-01 19:45:54 +02:00
1b7fe8fbee Use "options" in Gather instead of "progress"
Also made it possible to toggle Fill's toggle through parameters
2013-09-29 15:32:58 +02:00
a5a327388a Add ability not to fetch an item's article 2013-09-25 13:47:05 +02:00
0657077191 Add support for twitter
Grabs "feed" from the html page, clips tweet and article together.
2013-09-25 12:37:14 +02:00
da14242bcf Add feedify, and use it in morss 2013-09-25 12:36:21 +02:00
9bc4417be3 More flexible xml caching
New includes a 'type' var, to remember what we did out of it (normal, nothing, grabbed xml link, etc). xml/html mimetype are now saved in a dict, for easier editing, and consistency.
2013-09-25 12:32:40 +02:00
edff54a016 Add pushContent in feeds.py
Useful for twitter (later) for it's "clip" toggle, which keeps the original desc/content above the article. Makes changing the content, while keeping the original stub in place, easier.
2013-09-25 12:18:22 +02:00
208d70d3db Use separate var in Fill for final url
That way the url can be changed altogether for the article-fetching part, without changing the item link itself. Useful for upcoming twitter feeds.
2013-09-25 11:51:48 +02:00
fd1501a0c0 Check relative url earlier 2013-09-25 11:49:45 +02:00
1e621099e0 Log cache hash in Gather 2013-09-25 11:15:11 +02:00
3d6d7e70b6 Remove useless "as" in error catch 2013-09-25 11:14:22 +02:00
e73cbf56c2 Add 'html' option, usefull to see error on server 2013-09-25 11:13:33 +02:00
03014a8cbf Typo in UA_HTML var name 2013-09-25 11:11:11 +02:00
4a5cbcfd18 Move httplib in common code
Needed for error catch
2013-09-25 11:10:16 +02:00
3fd34ff1a6 decodeHTML works without connection object 2013-09-25 11:08:58 +02:00
658f51e5a9 Support feeds handed out as text/html
<http://www.pro-linux.de/rss/index1.xml> and <http://tehrantimes.com/index.php?option=com_ninjarsssyndicator&feed_id=1&format=raw> are on an equal footing…
2013-09-16 00:33:24 +02:00