Mercurial > hg > Feedworm

annotate backend/sqlalchemy/SqlAlchemyBackend.py @ 206:f74fe7cb5091

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
when updating feeds, only ever create new Feed objects for entries that are younger than the current expire date. This ensures that we do not see old, read, expired entries again
author dirk
date Sat, 02 Jun 2012 04:30:04 +0200
parents 7157bfd3f4d4
children bb3c851b18b1
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
119
04a730f9d07d move all sqlalchemy related classes to the respective sub-package. use a backend to abstract from access to the data
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
1
145
71c5dc02ff87 move the code from FeedList into the backend class
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 128
diff changeset
2 from Feed import Feed
71c5dc02ff87 move the code from FeedList into the backend class
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 128
diff changeset
3 from FeedEntry import FeedEntry
168
f4708d38419c move methods around so the SqlAlchemyBackend only needs to import the class
dirk
parents: 167
diff changeset
4 from FeedUpdater import FeedUpdater
119
04a730f9d07d move all sqlalchemy related classes to the respective sub-package. use a backend to abstract from access to the data
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
5 from Preferences import Preferences
155
a05719a6175e move common functionality into an abstract backend class, have both backends inherit from it. Implement enough of the couchdb backend that reading feeds (and marking feed entries as read) is possible
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 154
diff changeset
6 from backend.AbstractBackend import AbstractBackend
145
71c5dc02ff87 move the code from FeedList into the backend class
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 128
diff changeset
7 from sqlalchemy.orm import joinedload
128
32a173cb081c move updating the feeds to the backend
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 126
diff changeset
8 from sqlalchemy.sql import and_
119
04a730f9d07d move all sqlalchemy related classes to the respective sub-package. use a backend to abstract from access to the data
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
9 import Database
128
32a173cb081c move updating the feeds to the backend
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 126
diff changeset
10 import Mapping
119
04a730f9d07d move all sqlalchemy related classes to the respective sub-package. use a backend to abstract from access to the data
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
11 import logging
04a730f9d07d move all sqlalchemy related classes to the respective sub-package. use a backend to abstract from access to the data
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
12 import util
04a730f9d07d move all sqlalchemy related classes to the respective sub-package. use a backend to abstract from access to the data
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
13
155
a05719a6175e move common functionality into an abstract backend class, have both backends inherit from it. Implement enough of the couchdb backend that reading feeds (and marking feed entries as read) is possible
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 154
diff changeset
14 class SqlAlchemyBackend(AbstractBackend):
119
04a730f9d07d move all sqlalchemy related classes to the respective sub-package. use a backend to abstract from access to the data
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
15 '''
04a730f9d07d move all sqlalchemy related classes to the respective sub-package. use a backend to abstract from access to the data
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
16 Backend that uses sqlalchemy for persistence
04a730f9d07d move all sqlalchemy related classes to the respective sub-package. use a backend to abstract from access to the data
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
17 '''
04a730f9d07d move all sqlalchemy related classes to the respective sub-package. use a backend to abstract from access to the data
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
18
04a730f9d07d move all sqlalchemy related classes to the respective sub-package. use a backend to abstract from access to the data
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
19 def __init__(self):
155
a05719a6175e move common functionality into an abstract backend class, have both backends inherit from it. Implement enough of the couchdb backend that reading feeds (and marking feed entries as read) is possible
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 154
diff changeset
20 AbstractBackend.__init__(self)
119
04a730f9d07d move all sqlalchemy related classes to the respective sub-package. use a backend to abstract from access to the data
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
21 self._initLogging()
04a730f9d07d move all sqlalchemy related classes to the respective sub-package. use a backend to abstract from access to the data
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
22 self.session = Database.createSession()
04a730f9d07d move all sqlalchemy related classes to the respective sub-package. use a backend to abstract from access to the data
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
23 self.prefs = Preferences(self.session)
121
510a5d00e98a re-enabled AddFeed - does not work yet
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 119
diff changeset
24 self.updater = None
119
04a730f9d07d move all sqlalchemy related classes to the respective sub-package. use a backend to abstract from access to the data
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
25
04a730f9d07d move all sqlalchemy related classes to the respective sub-package. use a backend to abstract from access to the data
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
26 def _initLogging(self):
04a730f9d07d move all sqlalchemy related classes to the respective sub-package. use a backend to abstract from access to the data
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
27 logging.getLogger("sqlalchemy.orm").setLevel(logging.WARN)
04a730f9d07d move all sqlalchemy related classes to the respective sub-package. use a backend to abstract from access to the data
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
28
04a730f9d07d move all sqlalchemy related classes to the respective sub-package. use a backend to abstract from access to the data
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
29 sqlalchemyLogLevel = logging.ERROR
04a730f9d07d move all sqlalchemy related classes to the respective sub-package. use a backend to abstract from access to the data
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
30 if util.databaseLoggingEnabled():
04a730f9d07d move all sqlalchemy related classes to the respective sub-package. use a backend to abstract from access to the data
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
31 sqlalchemyLogLevel = logging.INFO
04a730f9d07d move all sqlalchemy related classes to the respective sub-package. use a backend to abstract from access to the data
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
32 logging.getLogger("sqlalchemy").setLevel(sqlalchemyLogLevel)
04a730f9d07d move all sqlalchemy related classes to the respective sub-package. use a backend to abstract from access to the data
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
33
04a730f9d07d move all sqlalchemy related classes to the respective sub-package. use a backend to abstract from access to the data
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
34 def preferences(self):
04a730f9d07d move all sqlalchemy related classes to the respective sub-package. use a backend to abstract from access to the data
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
35 return self.prefs
04a730f9d07d move all sqlalchemy related classes to the respective sub-package. use a backend to abstract from access to the data
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
36
154
2682db280be0 reorder methods
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 153
diff changeset
37 def dispose(self):
2682db280be0 reorder methods
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 153
diff changeset
38 # save all uncommitted state, just in case
2682db280be0 reorder methods
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 153
diff changeset
39 self.session.commit()
2682db280be0 reorder methods
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 153
diff changeset
40 self.session.close()
2682db280be0 reorder methods
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 153
diff changeset
41
153
65c4bb6d5add move management of the selected feed entry into the backend - sqlachemy backend works, couchdb backend currently broken
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 151
diff changeset
42 #
65c4bb6d5add move management of the selected feed entry into the backend - sqlachemy backend works, couchdb backend currently broken
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 151
diff changeset
43 # handling of feeds
65c4bb6d5add move management of the selected feed entry into the backend - sqlachemy backend works, couchdb backend currently broken
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 151
diff changeset
44 #
65c4bb6d5add move management of the selected feed entry into the backend - sqlachemy backend works, couchdb backend currently broken
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 151
diff changeset
45
119
04a730f9d07d move all sqlalchemy related classes to the respective sub-package. use a backend to abstract from access to the data
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
46 def getFeeds(self):
145
71c5dc02ff87 move the code from FeedList into the backend class
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 128
diff changeset
47 if self.preferences().showOnlyUnreadFeeds():
151
bca9341dc67f move the selected feed into the backend - sqlalchemy backend works, couchdb backend currently broken
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 146
diff changeset
48 self.feeds = self._getUnreadFeeds()
145
71c5dc02ff87 move the code from FeedList into the backend class
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 128
diff changeset
49 else:
151
bca9341dc67f move the selected feed into the backend - sqlalchemy backend works, couchdb backend currently broken
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 146
diff changeset
50 self.feeds = Feed.all(self.session)
bca9341dc67f move the selected feed into the backend - sqlalchemy backend works, couchdb backend currently broken
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 146
diff changeset
51 return self.feeds
145
71c5dc02ff87 move the code from FeedList into the backend class
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 128
diff changeset
52
71c5dc02ff87 move the code from FeedList into the backend class
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 128
diff changeset
53 def _getUnreadFeeds(self):
71c5dc02ff87 move the code from FeedList into the backend class
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 128
diff changeset
54 query = self.session.query(FeedEntry).filter(FeedEntry.read == 0)
71c5dc02ff87 move the code from FeedList into the backend class
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 128
diff changeset
55 queryWithOptions = query.options(joinedload("feed"))
71c5dc02ff87 move the code from FeedList into the backend class
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 128
diff changeset
56 result = queryWithOptions.all()
71c5dc02ff87 move the code from FeedList into the backend class
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 128
diff changeset
57 return self._collectFeeds(result)
71c5dc02ff87 move the code from FeedList into the backend class
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 128
diff changeset
58
71c5dc02ff87 move the code from FeedList into the backend class
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 128
diff changeset
59 def _collectFeeds(self, feedEntries):
71c5dc02ff87 move the code from FeedList into the backend class
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 128
diff changeset
60 feeds = [entry.feed for entry in feedEntries]
71c5dc02ff87 move the code from FeedList into the backend class
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 128
diff changeset
61 uniqueFeeds = set(feeds)
71c5dc02ff87 move the code from FeedList into the backend class
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 128
diff changeset
62 return list(uniqueFeeds)
119
04a730f9d07d move all sqlalchemy related classes to the respective sub-package. use a backend to abstract from access to the data
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
63
155
a05719a6175e move common functionality into an abstract backend class, have both backends inherit from it. Implement enough of the couchdb backend that reading feeds (and marking feed entries as read) is possible
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 154
diff changeset
64 def _retrieveEntriesForSelectedFeed(self, hideReadEntries):
a05719a6175e move common functionality into an abstract backend class, have both backends inherit from it. Implement enough of the couchdb backend that reading feeds (and marking feed entries as read) is possible
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 154
diff changeset
65 return self.selectedFeed.entriesSortedByUpdateDate(hideReadEntries)
151
bca9341dc67f move the selected feed into the backend - sqlalchemy backend works, couchdb backend currently broken
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 146
diff changeset
66
bca9341dc67f move the selected feed into the backend - sqlalchemy backend works, couchdb backend currently broken
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 146
diff changeset
67 def deleteSelectedFeed(self):
bca9341dc67f move the selected feed into the backend - sqlalchemy backend works, couchdb backend currently broken
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 146
diff changeset
68 self.session.delete(self.selectedFeed)
bca9341dc67f move the selected feed into the backend - sqlalchemy backend works, couchdb backend currently broken
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 146
diff changeset
69 self.session.commit()
bca9341dc67f move the selected feed into the backend - sqlalchemy backend works, couchdb backend currently broken
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 146
diff changeset
70
154
2682db280be0 reorder methods
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 153
diff changeset
71 def createFeed(self, url):
2682db280be0 reorder methods
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 153
diff changeset
72 try:
167
a3c945ce434c adjust the sqlalchemy backend to the changes in AbstractFeedUpdater
dirk
parents: 155
diff changeset
73 newFeed = Feed(url)
a3c945ce434c adjust the sqlalchemy backend to the changes in AbstractFeedUpdater
dirk
parents: 155
diff changeset
74 self.session.add(newFeed)
a3c945ce434c adjust the sqlalchemy backend to the changes in AbstractFeedUpdater
dirk
parents: 155
diff changeset
75
168
f4708d38419c move methods around so the SqlAlchemyBackend only needs to import the class
dirk
parents: 167
diff changeset
76 FeedUpdater(self.preferences(), self.session).update(newFeed)
154
2682db280be0 reorder methods
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 153
diff changeset
77 self.session.commit()
2682db280be0 reorder methods
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 153
diff changeset
78 except AttributeError as ae:
2682db280be0 reorder methods
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 153
diff changeset
79 self.session.rollback()
2682db280be0 reorder methods
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 153
diff changeset
80 raise ae
2682db280be0 reorder methods
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 153
diff changeset
81
2682db280be0 reorder methods
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 153
diff changeset
82 def updateAllFeeds(self):
167
a3c945ce434c adjust the sqlalchemy backend to the changes in AbstractFeedUpdater
dirk
parents: 155
diff changeset
83 FeedUpdater.updateAllFeeds(self.preferences(), self.session)
154
2682db280be0 reorder methods
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 153
diff changeset
84 self.session.commit()
2682db280be0 reorder methods
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 153
diff changeset
85
2682db280be0 reorder methods
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 153
diff changeset
86 def updateFeed(self, feed, changes):
2682db280be0 reorder methods
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 153
diff changeset
87 feed.takeChangesFrom(changes)
2682db280be0 reorder methods
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 153
diff changeset
88 feed.incrementNextUpdateDate()
2682db280be0 reorder methods
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 153
diff changeset
89 self.session.commit()
2682db280be0 reorder methods
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 153
diff changeset
90
153
65c4bb6d5add move management of the selected feed entry into the backend - sqlachemy backend works, couchdb backend currently broken
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 151
diff changeset
91 #
65c4bb6d5add move management of the selected feed entry into the backend - sqlachemy backend works, couchdb backend currently broken
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 151
diff changeset
92 # handling of the selected feed entry
65c4bb6d5add move management of the selected feed entry into the backend - sqlachemy backend works, couchdb backend currently broken
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 151
diff changeset
93 #
155
a05719a6175e move common functionality into an abstract backend class, have both backends inherit from it. Implement enough of the couchdb backend that reading feeds (and marking feed entries as read) is possible
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 154
diff changeset
94
a05719a6175e move common functionality into an abstract backend class, have both backends inherit from it. Implement enough of the couchdb backend that reading feeds (and marking feed entries as read) is possible
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 154
diff changeset
95 def _markSelectedFeedEntryRead(self):
153
65c4bb6d5add move management of the selected feed entry into the backend - sqlachemy backend works, couchdb backend currently broken
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 151
diff changeset
96 self.selectedFeedEntry.markRead()
65c4bb6d5add move management of the selected feed entry into the backend - sqlachemy backend works, couchdb backend currently broken
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 151
diff changeset
97
65c4bb6d5add move management of the selected feed entry into the backend - sqlachemy backend works, couchdb backend currently broken
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 151
diff changeset
98 def markFeedEntriesAsRead(self, indices):
65c4bb6d5add move management of the selected feed entry into the backend - sqlachemy backend works, couchdb backend currently broken
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 151
diff changeset
99 for index in indices:
65c4bb6d5add move management of the selected feed entry into the backend - sqlachemy backend works, couchdb backend currently broken
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 151
diff changeset
100 self.feedEntries[index].markRead()
65c4bb6d5add move management of the selected feed entry into the backend - sqlachemy backend works, couchdb backend currently broken
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 151
diff changeset
101 self.session.commit()
65c4bb6d5add move management of the selected feed entry into the backend - sqlachemy backend works, couchdb backend currently broken
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 151
diff changeset
102
65c4bb6d5add move management of the selected feed entry into the backend - sqlachemy backend works, couchdb backend currently broken
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 151
diff changeset
103 def toggleSelectedFeedEntryRead(self):
65c4bb6d5add move management of the selected feed entry into the backend - sqlachemy backend works, couchdb backend currently broken
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 151
diff changeset
104 self.selectedFeedEntry.toggleRead()
65c4bb6d5add move management of the selected feed entry into the backend - sqlachemy backend works, couchdb backend currently broken
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 151
diff changeset
105 self.session.commit()
151
bca9341dc67f move the selected feed into the backend - sqlalchemy backend works, couchdb backend currently broken
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 146
diff changeset
106
128
32a173cb081c move updating the feeds to the backend
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 126
diff changeset
107 def expireFeedEntries(self):
32a173cb081c move updating the feeds to the backend
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 126
diff changeset
108 logger = logging.getLogger("feedupdater")
32a173cb081c move updating the feeds to the backend
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 126
diff changeset
109 expireDate = self._calculateExpireDate()
32a173cb081c move updating the feeds to the backend
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 126
diff changeset
110 logger.info("expiring entries older than " + str(expireDate))
32a173cb081c move updating the feeds to the backend
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 126
diff changeset
111 feedEntry = Mapping.feedEntryTable
32a173cb081c move updating the feeds to the backend
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 126
diff changeset
112 deleteStatement = feedEntry.delete().where(
32a173cb081c move updating the feeds to the backend
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 126
diff changeset
113 and_(feedEntry.c.create_timestamp < expireDate, feedEntry.c.read == 1)
32a173cb081c move updating the feeds to the backend
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 126
diff changeset
114 )
32a173cb081c move updating the feeds to the backend
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 126
diff changeset
115 deleteStatement.execute()
32a173cb081c move updating the feeds to the backend
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 126
diff changeset
116 self.session.commit()