Mercurial > hg > Feedworm

annotate backend/sqlalchemy/FeedUpdater.py @ 215:b7caf487bc54

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
run make prior to starting the worm
author Dirk Olmes <dirk@xanthippe.ping.de>
date Mon, 13 May 2013 17:35:11 +0200
parents f4708d38419c
children bb3c851b18b1
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
4
e0199f383442 retrieve a feed for the given URL, store entries as feed_entry rows into the database
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
1
167
a3c945ce434c adjust the sqlalchemy backend to the changes in AbstractFeedUpdater
dirk
parents: 160
diff changeset
2 from Feed import Feed
a3c945ce434c adjust the sqlalchemy backend to the changes in AbstractFeedUpdater
dirk
parents: 160
diff changeset
3 from FeedEntry import FeedEntry
141
6ea813cfac33 pull out common code for updating a feed into an abstract class, have the sqlalchemy backend use that class.
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 123
diff changeset
4 from backend.AbstractFeedUpdater import AbstractFeedUpdater, FeedUpdateException
5
bfd47f55d85b add the updated date of the feed
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 4
diff changeset
5 from datetime import datetime
11
e87c54b3a216 use the logging framework for printing messages
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 10
diff changeset
6 import logging
4
e0199f383442 retrieve a feed for the given URL, store entries as feed_entry rows into the database
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
7
28
72dfae865899 better logging when updating feeds, handle entries that have no id
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 27
diff changeset
8 log = logging.getLogger("FeedUpdater")
9
fd4c8bfa62d6 FeedUpdater throws an exception if the URL could not be retrieved successfully. Includes unit tests.
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 7
diff changeset
9
168
f4708d38419c move methods around so the SqlAlchemyBackend only needs to import the class
dirk
parents: 167
diff changeset
10 def _findFeedsToUpdate(session):
35
aaec263f07ca Feeds manage the point in time when the next update should happen. FeedUpdater only updates feeds that are due.
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 28
diff changeset
11 return session.query(Feed).filter(Feed.next_update < datetime.now())
aaec263f07ca Feeds manage the point in time when the next update should happen. FeedUpdater only updates feeds that are due.
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 28
diff changeset
12
168
f4708d38419c move methods around so the SqlAlchemyBackend only needs to import the class
dirk
parents: 167
diff changeset
13 class FeedUpdater(AbstractFeedUpdater):
f4708d38419c move methods around so the SqlAlchemyBackend only needs to import the class
dirk
parents: 167
diff changeset
14 @staticmethod
f4708d38419c move methods around so the SqlAlchemyBackend only needs to import the class
dirk
parents: 167
diff changeset
15 def updateAllFeeds(preferences, session):
f4708d38419c move methods around so the SqlAlchemyBackend only needs to import the class
dirk
parents: 167
diff changeset
16 allFeeds = _findFeedsToUpdate(session)
f4708d38419c move methods around so the SqlAlchemyBackend only needs to import the class
dirk
parents: 167
diff changeset
17 for feed in allFeeds:
f4708d38419c move methods around so the SqlAlchemyBackend only needs to import the class
dirk
parents: 167
diff changeset
18 try:
f4708d38419c move methods around so the SqlAlchemyBackend only needs to import the class
dirk
parents: 167
diff changeset
19 FeedUpdater(preferences, session).update(feed)
f4708d38419c move methods around so the SqlAlchemyBackend only needs to import the class
dirk
parents: 167
diff changeset
20 except FeedUpdateException, fue:
f4708d38419c move methods around so the SqlAlchemyBackend only needs to import the class
dirk
parents: 167
diff changeset
21 log.warn("problems while updating feed " + feed.rss_url + ": " + str(fue))
f4708d38419c move methods around so the SqlAlchemyBackend only needs to import the class
dirk
parents: 167
diff changeset
22 session.commit()
123
862760b161b4 restructured adding a feed so that only the URL is passed into the backend - the rest of the operation is backend-internal
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 121
diff changeset
23
167
a3c945ce434c adjust the sqlalchemy backend to the changes in AbstractFeedUpdater
dirk
parents: 160
diff changeset
24 def __init__(self, preferences, session):
a3c945ce434c adjust the sqlalchemy backend to the changes in AbstractFeedUpdater
dirk
parents: 160
diff changeset
25 AbstractFeedUpdater.__init__(self, preferences)
4
e0199f383442 retrieve a feed for the given URL, store entries as feed_entry rows into the database
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
26 self.session = session
e0199f383442 retrieve a feed for the given URL, store entries as feed_entry rows into the database
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
27
141
6ea813cfac33 pull out common code for updating a feed into an abstract class, have the sqlalchemy backend use that class.
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 123
diff changeset
28 def _processEntry(self, entry):
4
e0199f383442 retrieve a feed for the given URL, store entries as feed_entry rows into the database
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
29 feedEntry = FeedEntry.findById(entry.id, self.session)
e0199f383442 retrieve a feed for the given URL, store entries as feed_entry rows into the database
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
30 if feedEntry is None:
141
6ea813cfac33 pull out common code for updating a feed into an abstract class, have the sqlalchemy backend use that class.
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 123
diff changeset
31 self._createFeedEntry(entry)
100
99807963d9e0 use the URL as feed title if the feed itself does not come with a title
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 85
diff changeset
32
141
6ea813cfac33 pull out common code for updating a feed into an abstract class, have the sqlalchemy backend use that class.
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 123
diff changeset
33 def _createFeedEntry(self, entry):
62
abc0516a1c0c FeedEntry provides a static method for creating new entries: better modularization and support for working with the class in interactive mode. FeedUpdater's normalize method is a module function now, again for ease of use in interactive scenarios
dirk@xanthippe.ping.de
parents: 58
diff changeset
34 new = FeedEntry.create(entry)
5
bfd47f55d85b add the updated date of the feed
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 4
diff changeset
35 new.feed = self.feed
bfd47f55d85b add the updated date of the feed
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 4
diff changeset
36 self.session.add(new)
66
97c4e94f99cf log when creating a new FeedEntry
dirk@xanthippe.ping.de
parents: 62
diff changeset
37 log.info("new feed entry: " + entry.title)
144
74217db92993 updating feeds on the couchdb backend works now
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 143
diff changeset
38
74217db92993 updating feeds on the couchdb backend works now
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 143
diff changeset
39 def _incrementFeedUpdateDate(self):
74217db92993 updating feeds on the couchdb backend works now
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 143
diff changeset
40 self.feed.incrementNextUpdateDate()