pictuga
/
morss
1
1
Fork 0
Code Issues Pull Requests Actions Packages Releases Activity
971 Commits
1 Branch
2 Tags
3.5 MiB
Commit Graph

543 Commits (d07aa566ed7fabd84965dd657674310f735e8c23)

Author SHA1 Message Date
pictuga 4ccc0dafcd Basic help for sub-lib interactive use 2020-05-26 19:34:20 +02:00
pictuga 2fe3e0b8ee feeds: clean up other stylesheets before putting ours 2020-05-26 19:26:36 +02:00
pictuga 68c46a1823 morss: remove deprecated twitter/fb link handling 2020-05-13 12:31:09 +02:00
pictuga 91be2d229e morss: ability to use first link from desc instead of default link 2020-05-13 12:29:53 +02:00
pictuga 038f267ea2 Rename :theforce into :force 2020-05-13 11:49:15 +02:00
pictuga 22005065e8 Use etree.tostring 'method' arg 
Gives appropriately formatted html code.
Some pages might otherwise be rendered as blank.
 2020-05-13 11:44:34 +02:00
pictuga 7d0d416610 morss: cache articles for 24hrs 
Also make it possible to refetch articles, regardless of cache
 2020-05-12 21:10:31 +02:00
pictuga 5dac4c69a1 crawler: more code comments 2020-05-12 20:44:25 +02:00
pictuga 36e2a1c3fd crawler: increase size limit from 100KiB to 500 
I'm looking at you, worldbankgroup.csod.com/ats/careersite/search.aspx
 2020-05-12 19:34:16 +02:00
pictuga 83dd2925d3 readabilite: better parsing 
Keeping blank_text keeps the tree more as-it, making the final output closer to expectations
 2020-05-12 14:15:53 +02:00
pictuga e09d0abf54 morss: remove deprecated peace of code 2020-05-07 16:05:30 +02:00
pictuga ff26a560cb Shift safari work around to morss.py 2020-05-07 16:04:54 +02:00
pictuga f685139137 crawler: use UPSERT statements 
Avoid potential race conditions
 2020-05-03 21:27:45 +02:00
pictuga 73b477665e morss: separate :clip with <hr> instead of stars 2020-05-02 19:19:54 +02:00
pictuga b425992783 morss: don't follow alt=rss with custom feeds 
To have the same page as with :get=page and to avoid shitty feeds
 2020-05-02 19:18:58 +02:00
pictuga 271ac8f80f crawler: comment code a bit 2020-05-02 19:18:01 +02:00
pictuga 64e41b807d crawler: handle http:/ (single slash) 
Fixing one more corner case! malayalam.oneindia.com
 2020-05-02 19:17:15 +02:00
pictuga 27a42c47aa morss: use final request url 
Code is not very elegant...
 2020-04-28 22:30:21 +02:00
pictuga c27c38f7c7 crawler: return dict instead of tuple 2020-04-28 22:29:07 +02:00
pictuga a1dc96cb50 feeds: remove mimetype from function call as no longer used 2020-04-28 22:07:25 +02:00
pictuga 749acc87fc Centralize url clean up in crawler.py 2020-04-28 22:03:49 +02:00
pictuga cb69e3167f crawler: accept non-ascii urls 
Covering one more corner case!
 2020-04-28 14:47:23 +02:00
pictuga c3f06da947 morss: process(): specify encoding for clarity 2020-04-28 14:45:00 +02:00
pictuga 44a3e0edc4 readabilite: specify in- and out-going encoding 2020-04-28 14:44:35 +02:00
pictuga 818cdaaa9b Make it possible to call sub-libs in non interactive mode 
Run `python -m morss.feeds http://lemonde.fr` and so on
 2020-04-27 18:00:14 +02:00
pictuga 2806c64326 Make it possible to directly run sub-libs (feeds, crawler, readabilite) 
Run `python -im morss.feeds http://website.sample/rss.xml` and so on
 2020-04-27 17:19:31 +02:00
pictuga f6bc23927f readabilite: drop dangerous tags (script, style) 2020-04-25 12:25:02 +02:00
pictuga c86572374e readabilite: minimum score requirement 2020-04-25 12:24:36 +02:00
pictuga 59ef5af9e2 feeds: fix bug when deleting attr in html 2020-04-24 22:12:05 +02:00
pictuga 6a0531ca03 crawler: randomize user agent 2020-04-24 11:28:39 +02:00
pictuga 8187876a06 crawler: stop at first alternative link 
Should save a few ms and the first one is usually (?) the most relevant/generic
 2020-04-23 11:23:45 +02:00
pictuga 325a373e3e feeds: add SyntaxError catch 2020-04-20 16:15:15 +02:00
pictuga 2719bd6776 crawler: fix chinese encoding 2020-04-20 16:14:55 +02:00
pictuga ec8edb02f1 Various small bug fixes 2020-04-19 12:54:02 +02:00
pictuga d01b943597 Remove leftover threading var 2020-04-19 12:51:11 +02:00
pictuga b361aa2867 Add timeout to :get 2020-04-19 12:50:26 +02:00
pictuga 4ce3c7cb32 Small code clean ups 2020-04-19 12:50:05 +02:00
pictuga 7e45b2611d Disable multi-threading 
Impact was mostly negative due to locks
 2020-04-19 12:29:52 +02:00
pictuga 036e5190f1 crawler: remove unused code 2020-04-18 21:40:02 +02:00
pictuga e99c5b3b71 morss: more sensible default MAX/LIM values 2020-04-18 17:21:45 +02:00
pictuga 7375adce33 sheet.xsl: fix & improve 2020-04-15 23:34:28 +02:00
pictuga fe82b19c91 Merge .xsl & html template 
Turns out they somehow serve a similar purpose
 2020-04-15 22:30:45 +02:00
pictuga 0b31e97492 morss: remove debug code in http file handler 2020-04-14 23:20:03 +02:00
pictuga b0ad7c259d Add README & LICENSE to data_files 2020-04-14 19:34:12 +02:00
pictuga 59139272fd Auto-detect the location of www/ 
Either ../www or /usr/share/morss
Adapted README accordingly
 2020-04-14 18:07:19 +02:00
pictuga e6b7c0eb33 Fix app definition for uwsgi 2020-04-13 15:30:09 +02:00
pictuga 67c096ad5b feeds: add fake path to default html parser 
Without it, some websites were accidentally matching it (false positives)
 2020-04-12 13:00:56 +02:00
pictuga f018437544 crawler: make mysql backend thread safe 2020-04-12 12:53:05 +02:00
pictuga 8e5e8d24a4 Timezone fixes 2020-04-10 20:33:59 +02:00
pictuga ee78a7875a morss: focus on the most recent feed items 2020-04-10 16:08:13 +02:00
pictuga 9e7b9d95ee feeds: properly use html template 2020-04-09 20:00:51 +02:00
pictuga 987a719c4e feeds: try all parsers regardless of contenttype 
Turns out some websites send the wrong contenttype (json for html, html for xml, etc.)
 2020-04-09 19:17:51 +02:00
pictuga 47b33f4baa morss: specify server output encoding 2020-04-09 19:10:45 +02:00
pictuga 3c7f512583 feeds: handle several errors 2020-04-09 19:09:10 +02:00
pictuga a32f5a8536 readabilite: add debug option (also used by :get) 2020-04-09 19:08:13 +02:00
pictuga 63a06524b7 morss: various encoding fixes 2020-04-09 19:06:51 +02:00
pictuga b0f80c6d3c morss: fix csv output encoding 2020-04-09 19:05:50 +02:00
pictuga 78cea10ead morss: replace :getpage with :get 
Also provides readabilite debugging
 2020-04-09 18:43:20 +02:00
pictuga e5a82ff1f4 crawler: drop auto-referer 
Was solving some issues. But creating even more issues.
 2020-04-07 10:39:21 +02:00
pictuga f3d1f92b39 Detect encoding everytime 2020-04-07 10:38:36 +02:00
pictuga 7691df5257 Use wrapper for http calls 2020-04-07 10:30:17 +02:00
pictuga f1d0431e68 morss: drop :html, replaced with :reader 
README updated accordingly
 2020-04-07 09:23:29 +02:00
pictuga a09831415f feeds: fix bug when mimetype matches nothing 2020-04-06 18:53:07 +02:00
pictuga bfad6b7a4a readabilite: clean before counting 
To remove links which are not kept anyway
 2020-04-06 16:55:39 +02:00
pictuga 6b8c3e51e7 readabilite: fix threshold feature 
Awkward typo...
 2020-04-06 16:52:06 +02:00
pictuga dc9e425247 readabilite: don't clean-out the top 10% nodes 
Loosen up the code once again to limit over-kill
 2020-04-06 14:26:28 +02:00
pictuga 2f48e18bb1 readabilite: put scores directly in html node 
Probably slower but makes code somewhat cleaner...
 2020-04-06 14:21:41 +02:00
pictuga a82ec96eb7 Delete feedify.py leftover code 
iTunes integration untested, unreliable and not working...
 2020-04-05 22:16:52 +02:00
pictuga aad2398e69 feeds: turns out lxml.etree doesn't have drop_tag 2020-04-05 21:50:38 +02:00
pictuga eeac630855 crawler: add more "realistic" headers 2020-04-05 21:11:57 +02:00
pictuga e136b0feb2 readabilite: loosen the slayer 
Previous impl. lead to too many empty results
 2020-04-05 20:47:30 +02:00
pictuga 6cf32af6c0 readabilite: also use BS 2020-04-05 20:46:42 +02:00
pictuga 568e7d7dd2 feeds: make BS's output bytes for lxml's sake 2020-04-05 20:46:04 +02:00
pictuga 3617f86e9d morss: make cgi_encore more robust 2020-04-05 16:43:11 +02:00
pictuga d90756b337 morss: drop 'keep' option 
Because the Firefox behaviour it is working around is no longer in use
 2020-04-05 16:37:27 +02:00
pictuga 40c69f17d2 feeds: parse html with BS 
More robust & to make it consistent with :getpage
 2020-04-05 16:12:41 +02:00
pictuga 99461ea185 crawler: fix var name issues (private_cache) 2020-04-05 16:11:36 +02:00
pictuga bf86c1e962 crawler: make AutoUA match http(s) type 2020-04-05 16:07:51 +02:00
pictuga d20f6237bd crawler: replace ContentNegoHandler with AlternateHandler 
More basic. Sends the same headers no matter what. Make requests more "replicable".
Also, drop "text/xml" from RSS contenttype, too broad, matches garbage
 2020-04-05 16:05:59 +02:00
pictuga 8a4d68d72c crawler: drop 'basic' toggle 
Can't even remember the use case
 2020-04-05 16:03:06 +02:00
pictuga e6811138fd morss: use redirected url in :getpage 
Still have to find how to do the same thing with feeds...
 2020-04-04 20:04:57 +02:00
pictuga 35b702fffd morss: default values for feed creation 2020-04-04 19:39:32 +02:00
pictuga 4a88886767 morss: get_page to act as a basic proxy (for iframes) 2020-04-04 16:37:15 +02:00
pictuga 1653394cf7 morss: cgi_dispatcher to be able to create extra functions 2020-04-04 16:35:16 +02:00
pictuga a8a90cf414 morss: move url/options parsing to own function 
For future re-use
 2020-04-04 16:33:52 +02:00
pictuga bdbaf0f8a7 morss/cgi: fix handling of special chars in url 2020-04-04 16:21:37 +02:00
pictuga d0e447a2a6 ItemFix: clean up Pocket links 2020-04-04 16:20:39 +02:00
pictuga 7c3091d64c morss: code spacing 
One of those commits that make me feel useful
 2020-03-21 23:41:46 +01:00
pictuga 37b4e144a9 morss: small fixes 
Includes dropping off ftp support
 2020-03-21 23:30:18 +01:00
pictuga bd4b7b5bb2 morss: convert HTML feeds to XML ones for completeness 2020-03-21 23:27:42 +01:00
pictuga 68d920d4b5 morss: make FeedFormat more flexible with encoding 2020-03-21 23:26:35 +01:00
pictuga 758ff404a8 morss: fix cgi_app silent output 
*Must* return sth
 2020-03-21 23:25:25 +01:00
pictuga 463530f02c morss: middleware to enforce encoding 
bytes are always expected
 2020-03-21 23:23:50 +01:00
pictuga ec0a28a91d morss: use middleware for wsgi apps 2020-03-21 23:23:21 +01:00
pictuga 421acb439d morss: make errors more readable over http 2020-03-21 23:08:29 +01:00
pictuga 42c5d09ccb morss: split "options" var into "raw_options" & "options" 
To make it clearer who-is-what
 2020-03-21 23:07:07 +01:00
pictuga 056de12484 morss: add sheet.xsl to file handled by http server 2020-03-21 23:06:28 +01:00
pictuga 961a31141f morss: fix url fixing 2020-03-21 17:28:00 +01:00
pictuga a7b01ee85e readabilite: further html processing instructions fix 2020-03-21 17:23:50 +01:00
pictuga 4d785820d9 feeds: ignore provided stylesheets and add ours 
Provided sheets usually create errors. Ours is (hopefully) more informative for users not familiar with RSS feeds
 2020-03-20 15:32:44 +01:00
pictuga 6a01fc439e feeds: better handle "empty" datetime 2020-03-20 12:30:42 +01:00
pictuga d24734110a morss: convert all feeds to RSS 
As html feeds might not contain some feeds, leading to data loss
 2020-03-20 12:26:34 +01:00
pictuga a41c2a3a62 morss: fix twitter link detection 2020-03-20 12:26:19 +01:00
pictuga dd2651061f feeds & morss: clean up comments/empty lines 2020-03-20 12:25:48 +01:00
pictuga 912c323c40 feeds: make function output more consistent 
e.g. setters return nothing, getters return sth relevant or None (i.e. no empty strings)
 2020-03-20 12:23:15 +01:00
pictuga 5705a0be17 feeds: fix delete/rmv code 2020-03-20 12:22:07 +01:00
pictuga 4735ffba45 feeds: fix .convert auto-convert 
To fix inheritance loophole
 2020-03-20 12:20:41 +01:00
pictuga 08e39f5631 feeds: give simpler name to helper functions 2020-03-20 12:20:15 +01:00
pictuga 765a43511e feeds: remove unused import 2020-03-20 12:19:08 +01:00
pictuga 5865af64f9 Fix indent output for html/xml 2020-03-20 12:18:13 +01:00
pictuga b3b90c067a morss.py: remove "useless" functions 
Have to keep the code clean
 2020-03-20 11:19:06 +01:00
pictuga 5288cc8796 Clean up unused import's 2020-03-19 15:09:53 +01:00
pictuga d12d44a500 Remove unused hash-bangs 
Leftovers from debugging
 2020-03-19 15:06:28 +01:00
pictuga ee8c57c1fc feeds: avoid convert to self 2020-03-19 12:54:04 +01:00
pictuga bda51b0fc7 feeds & morss: many encoding/tostring fixes 2020-03-19 12:53:25 +01:00
pictuga c09b457168 feeds: fix .dic code 
Bug introduced recently…
 2020-03-19 11:36:20 +01:00
pictuga b47e40246c feeds: clean up html code handling 2020-03-19 11:35:51 +01:00
pictuga 9cf933723f feeds: clean up time handling 
Includes a shameful fix on @property
 2020-03-19 11:35:02 +01:00
pictuga d26795dce8 morss: from feedify to feeds 
Also scrap obsolete feedify code
 2020-03-19 10:27:44 +01:00
pictuga 2704e91a3d readabilite: handle another weird html stuff 2020-03-19 10:24:09 +01:00
pictuga f48961a7e4 feeds: small code cleanups 2020-03-19 10:13:22 +01:00
pictuga fd51f74eb5 feeds: drop facebook-related code 
No longer maintained and of little use anyway
 2020-03-19 10:03:59 +01:00
pictuga d3fe51cea5 feeds: remove duplicated code 2020-03-19 09:49:52 +01:00
pictuga 449bc3c695 feeds: fix handling of html code 2020-03-19 09:48:53 +01:00
pictuga 13ea52ef80 feeds: add .torss() 2020-03-19 09:47:58 +01:00
pictuga aa2b56b266 feeds: small code cleanups 2020-03-19 09:47:17 +01:00
pictuga 296e9a6c13 feedifiy.ini: clean up following feeds.py's html intro 2020-03-18 16:52:28 +01:00
pictuga 9dbe061fd6 Remove markdown-related code 
Time to clean up the code and stop with those non-core features
They just make the code harder to maintain
 2020-03-18 16:47:00 +01:00
pictuga 4a70aa9dfa feeds: auto-parse() 2020-03-18 16:34:40 +01:00
pictuga c2f85da94a feeds: add html support, adapt .tohtml() 2020-03-18 16:33:10 +01:00
pictuga e3528a8f36 feeds: use FeddJSON for .tojson() 
Clean up related code
 2020-03-18 16:31:36 +01:00
pictuga 2dd9ae202d feeds: add 'mode' to Parsers 2020-03-18 16:24:08 +01:00
pictuga 8e3d32f24c feds: rename 'ruleset' into 'default_ruleset' 
Better reflects its use
 2020-03-18 16:22:03 +01:00
pictuga 6ce616106b feeds: disable 'multi' ruleset 
RSS ruleset has been cut into 4 rulesets
 2020-03-18 16:20:42 +01:00
pictuga 186fa2b408 feedify: remove empty 'path' lines 2020-03-18 16:18:22 +01:00
pictuga e9d46cb6a9 feeds: move mimetypes into .py from .ini 2020-03-18 16:08:42 +01:00
pictuga 7644c550ec feedify: remove id, is_permalink as well 2020-03-17 16:48:36 +01:00
pictuga e460fdc8f4 feeds: remove little-used properties 
id, is_permalink
Also a good opportunity to trash the unreliable bool-parsing code
 2020-03-17 16:46:42 +01:00
pictuga 1e714ab34b feeds: add ability to convert to another typo of feed 2020-03-17 14:02:24 +01:00
pictuga 4ba4d73ce6 feeds: add json support 2020-03-17 14:02:01 +01:00
pictuga 6d5aa8c222 feeds: clean up .append() 2020-03-17 13:59:51 +01:00
pictuga f10727f94a feeds: small code cleanup's 2020-03-17 12:26:34 +01:00
pictuga d42e19a165 feeds: beter rules handling 
"Dynamic" rule set picking, better handling of non-multi rules
 2020-03-17 12:23:36 +01:00
pictuga fe46c6c522 feeds: pass parent Feed to Items 2020-03-17 12:22:14 +01:00
pictuga 9c557ea02c feeds: fix function def 2020-03-17 11:08:40 +01:00
pictuga 8a4f86210c feedify.ini: remove utf-8 declaration 
Screws up with the parser as it is read as unicode (and xml parser expects bytes)
 2020-03-17 11:06:59 +01:00
pictuga ce30952fa2 feeds: make "rule" split clearer 
"rrule" var name to tell appart the original "rule" from the parsed one
 2020-03-16 17:46:04 +01:00
pictuga 3fb6ff891c feeds: share more code, add comments 
Should reduce redundancy
 2020-03-16 17:45:08 +01:00
pictuga f5acd2c14c feeds: use RawConfigParser 
This one does not try to replace non-std characters (e.g. %)
 2020-03-16 17:43:03 +01:00
pictuga 7cb3b29ef2 feeds: remove unused import 2020-03-16 17:38:48 +01:00