Mercurial > hg > Feedworm

annotate backend/sqlalchemy/SqlAlchemyBackend.py @ 259:304917762618 default tip

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
implementation of feed updates
author Dirk Olmes <dirk@xanthippe.ping.de>
date Tue, 12 Mar 2019 02:41:22 +0100
parents 8e73a8ae863f
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
217
bb3c851b18b1 add source file endcoding header
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 191
diff changeset
1 # -*- coding: utf-8 -*-
218
699d8f1cebd4 unify imports, especially Qt imports. Use consistent super syntax
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 217
diff changeset
2 import Database
699d8f1cebd4 unify imports, especially Qt imports. Use consistent super syntax
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 217
diff changeset
3 import Mapping
699d8f1cebd4 unify imports, especially Qt imports. Use consistent super syntax
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 217
diff changeset
4 import logging
699d8f1cebd4 unify imports, especially Qt imports. Use consistent super syntax
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 217
diff changeset
5 import util
145
71c5dc02ff87 move the code from FeedList into the backend class
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 128
diff changeset
6 from Feed import Feed
71c5dc02ff87 move the code from FeedList into the backend class
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 128
diff changeset
7 from FeedEntry import FeedEntry
168
f4708d38419c move methods around so the SqlAlchemyBackend only needs to import the class
dirk
parents: 167
diff changeset
8 from FeedUpdater import FeedUpdater
119
04a730f9d07d move all sqlalchemy related classes to the respective sub-package. use a backend to abstract from access to the data
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
9 from Preferences import Preferences
155
a05719a6175e move common functionality into an abstract backend class, have both backends inherit from it. Implement enough of the couchdb backend that reading feeds (and marking feed entries as read) is possible
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 154
diff changeset
10 from backend.AbstractBackend import AbstractBackend
145
71c5dc02ff87 move the code from FeedList into the backend class
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 128
diff changeset
11 from sqlalchemy.orm import joinedload
128
32a173cb081c move updating the feeds to the backend
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 126
diff changeset
12 from sqlalchemy.sql import and_
119
04a730f9d07d move all sqlalchemy related classes to the respective sub-package. use a backend to abstract from access to the data
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
13
245
8e73a8ae863f Fix the docstrings
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 218
diff changeset
14 """
8e73a8ae863f Fix the docstrings
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 218
diff changeset
15 Backend that uses sqlalchemy for persistence
8e73a8ae863f Fix the docstrings
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 218
diff changeset
16 """
155
a05719a6175e move common functionality into an abstract backend class, have both backends inherit from it. Implement enough of the couchdb backend that reading feeds (and marking feed entries as read) is possible
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 154
diff changeset
17 class SqlAlchemyBackend(AbstractBackend):
119
04a730f9d07d move all sqlalchemy related classes to the respective sub-package. use a backend to abstract from access to the data
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
18 def __init__(self):
155
a05719a6175e move common functionality into an abstract backend class, have both backends inherit from it. Implement enough of the couchdb backend that reading feeds (and marking feed entries as read) is possible
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 154
diff changeset
19 AbstractBackend.__init__(self)
119
04a730f9d07d move all sqlalchemy related classes to the respective sub-package. use a backend to abstract from access to the data
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
20 self._initLogging()
04a730f9d07d move all sqlalchemy related classes to the respective sub-package. use a backend to abstract from access to the data
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
21 self.session = Database.createSession()
04a730f9d07d move all sqlalchemy related classes to the respective sub-package. use a backend to abstract from access to the data
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
22 self.prefs = Preferences(self.session)
121
510a5d00e98a re-enabled AddFeed - does not work yet
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 119
diff changeset
23 self.updater = None
119
04a730f9d07d move all sqlalchemy related classes to the respective sub-package. use a backend to abstract from access to the data
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
24
04a730f9d07d move all sqlalchemy related classes to the respective sub-package. use a backend to abstract from access to the data
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
25 def _initLogging(self):
04a730f9d07d move all sqlalchemy related classes to the respective sub-package. use a backend to abstract from access to the data
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
26 logging.getLogger("sqlalchemy.orm").setLevel(logging.WARN)
04a730f9d07d move all sqlalchemy related classes to the respective sub-package. use a backend to abstract from access to the data
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
27
04a730f9d07d move all sqlalchemy related classes to the respective sub-package. use a backend to abstract from access to the data
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
28 sqlalchemyLogLevel = logging.ERROR
04a730f9d07d move all sqlalchemy related classes to the respective sub-package. use a backend to abstract from access to the data
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
29 if util.databaseLoggingEnabled():
04a730f9d07d move all sqlalchemy related classes to the respective sub-package. use a backend to abstract from access to the data
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
30 sqlalchemyLogLevel = logging.INFO
04a730f9d07d move all sqlalchemy related classes to the respective sub-package. use a backend to abstract from access to the data
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
31 logging.getLogger("sqlalchemy").setLevel(sqlalchemyLogLevel)
04a730f9d07d move all sqlalchemy related classes to the respective sub-package. use a backend to abstract from access to the data
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
32
04a730f9d07d move all sqlalchemy related classes to the respective sub-package. use a backend to abstract from access to the data
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
33 def preferences(self):
04a730f9d07d move all sqlalchemy related classes to the respective sub-package. use a backend to abstract from access to the data
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
34 return self.prefs
04a730f9d07d move all sqlalchemy related classes to the respective sub-package. use a backend to abstract from access to the data
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
35
154
2682db280be0 reorder methods
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 153
diff changeset
36 def dispose(self):
2682db280be0 reorder methods
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 153
diff changeset
37 # save all uncommitted state, just in case
2682db280be0 reorder methods
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 153
diff changeset
38 self.session.commit()
2682db280be0 reorder methods
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 153
diff changeset
39 self.session.close()
2682db280be0 reorder methods
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 153
diff changeset
40
153
65c4bb6d5add move management of the selected feed entry into the backend - sqlachemy backend works, couchdb backend currently broken
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 151
diff changeset
41 #
65c4bb6d5add move management of the selected feed entry into the backend - sqlachemy backend works, couchdb backend currently broken
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 151
diff changeset
42 # handling of feeds
65c4bb6d5add move management of the selected feed entry into the backend - sqlachemy backend works, couchdb backend currently broken
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 151
diff changeset
43 #
65c4bb6d5add move management of the selected feed entry into the backend - sqlachemy backend works, couchdb backend currently broken
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 151
diff changeset
44
119
04a730f9d07d move all sqlalchemy related classes to the respective sub-package. use a backend to abstract from access to the data
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
45 def getFeeds(self):
145
71c5dc02ff87 move the code from FeedList into the backend class
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 128
diff changeset
46 if self.preferences().showOnlyUnreadFeeds():
151
bca9341dc67f move the selected feed into the backend - sqlalchemy backend works, couchdb backend currently broken
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 146
diff changeset
47 self.feeds = self._getUnreadFeeds()
145
71c5dc02ff87 move the code from FeedList into the backend class
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 128
diff changeset
48 else:
151
bca9341dc67f move the selected feed into the backend - sqlalchemy backend works, couchdb backend currently broken
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 146
diff changeset
49 self.feeds = Feed.all(self.session)
bca9341dc67f move the selected feed into the backend - sqlalchemy backend works, couchdb backend currently broken
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 146
diff changeset
50 return self.feeds
145
71c5dc02ff87 move the code from FeedList into the backend class
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 128
diff changeset
51
71c5dc02ff87 move the code from FeedList into the backend class
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 128
diff changeset
52 def _getUnreadFeeds(self):
71c5dc02ff87 move the code from FeedList into the backend class
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 128
diff changeset
53 query = self.session.query(FeedEntry).filter(FeedEntry.read == 0)
71c5dc02ff87 move the code from FeedList into the backend class
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 128
diff changeset
54 queryWithOptions = query.options(joinedload("feed"))
71c5dc02ff87 move the code from FeedList into the backend class
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 128
diff changeset
55 result = queryWithOptions.all()
71c5dc02ff87 move the code from FeedList into the backend class
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 128
diff changeset
56 return self._collectFeeds(result)
71c5dc02ff87 move the code from FeedList into the backend class
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 128
diff changeset
57
71c5dc02ff87 move the code from FeedList into the backend class
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 128
diff changeset
58 def _collectFeeds(self, feedEntries):
71c5dc02ff87 move the code from FeedList into the backend class
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 128
diff changeset
59 feeds = [entry.feed for entry in feedEntries]
71c5dc02ff87 move the code from FeedList into the backend class
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 128
diff changeset
60 uniqueFeeds = set(feeds)
71c5dc02ff87 move the code from FeedList into the backend class
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 128
diff changeset
61 return list(uniqueFeeds)
119
04a730f9d07d move all sqlalchemy related classes to the respective sub-package. use a backend to abstract from access to the data
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
62
155
a05719a6175e move common functionality into an abstract backend class, have both backends inherit from it. Implement enough of the couchdb backend that reading feeds (and marking feed entries as read) is possible
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 154
diff changeset
63 def _retrieveEntriesForSelectedFeed(self, hideReadEntries):
a05719a6175e move common functionality into an abstract backend class, have both backends inherit from it. Implement enough of the couchdb backend that reading feeds (and marking feed entries as read) is possible
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 154
diff changeset
64 return self.selectedFeed.entriesSortedByUpdateDate(hideReadEntries)
151
bca9341dc67f move the selected feed into the backend - sqlalchemy backend works, couchdb backend currently broken
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 146
diff changeset
65
bca9341dc67f move the selected feed into the backend - sqlalchemy backend works, couchdb backend currently broken
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 146
diff changeset
66 def deleteSelectedFeed(self):
bca9341dc67f move the selected feed into the backend - sqlalchemy backend works, couchdb backend currently broken
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 146
diff changeset
67 self.session.delete(self.selectedFeed)
bca9341dc67f move the selected feed into the backend - sqlalchemy backend works, couchdb backend currently broken
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 146
diff changeset
68 self.session.commit()
bca9341dc67f move the selected feed into the backend - sqlalchemy backend works, couchdb backend currently broken
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 146
diff changeset
69
154
2682db280be0 reorder methods
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 153
diff changeset
70 def createFeed(self, url):
2682db280be0 reorder methods
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 153
diff changeset
71 try:
167
a3c945ce434c adjust the sqlalchemy backend to the changes in AbstractFeedUpdater
dirk
parents: 155
diff changeset
72 newFeed = Feed(url)
a3c945ce434c adjust the sqlalchemy backend to the changes in AbstractFeedUpdater
dirk
parents: 155
diff changeset
73 self.session.add(newFeed)
a3c945ce434c adjust the sqlalchemy backend to the changes in AbstractFeedUpdater
dirk
parents: 155
diff changeset
74
168
f4708d38419c move methods around so the SqlAlchemyBackend only needs to import the class
dirk
parents: 167
diff changeset
75 FeedUpdater(self.preferences(), self.session).update(newFeed)
154
2682db280be0 reorder methods
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 153
diff changeset
76 self.session.commit()
2682db280be0 reorder methods
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 153
diff changeset
77 except AttributeError as ae:
2682db280be0 reorder methods
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 153
diff changeset
78 self.session.rollback()
2682db280be0 reorder methods
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 153
diff changeset
79 raise ae
2682db280be0 reorder methods
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 153
diff changeset
80
2682db280be0 reorder methods
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 153
diff changeset
81 def updateAllFeeds(self):
167
a3c945ce434c adjust the sqlalchemy backend to the changes in AbstractFeedUpdater
dirk
parents: 155
diff changeset
82 FeedUpdater.updateAllFeeds(self.preferences(), self.session)
154
2682db280be0 reorder methods
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 153
diff changeset
83 self.session.commit()
2682db280be0 reorder methods
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 153
diff changeset
84
2682db280be0 reorder methods
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 153
diff changeset
85 def updateFeed(self, feed, changes):
2682db280be0 reorder methods
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 153
diff changeset
86 feed.takeChangesFrom(changes)
2682db280be0 reorder methods
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 153
diff changeset
87 feed.incrementNextUpdateDate()
2682db280be0 reorder methods
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 153
diff changeset
88 self.session.commit()
2682db280be0 reorder methods
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 153
diff changeset
89
153
65c4bb6d5add move management of the selected feed entry into the backend - sqlachemy backend works, couchdb backend currently broken
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 151
diff changeset
90 #
65c4bb6d5add move management of the selected feed entry into the backend - sqlachemy backend works, couchdb backend currently broken
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 151
diff changeset
91 # handling of the selected feed entry
65c4bb6d5add move management of the selected feed entry into the backend - sqlachemy backend works, couchdb backend currently broken
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 151
diff changeset
92 #
155
a05719a6175e move common functionality into an abstract backend class, have both backends inherit from it. Implement enough of the couchdb backend that reading feeds (and marking feed entries as read) is possible
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 154
diff changeset
93
a05719a6175e move common functionality into an abstract backend class, have both backends inherit from it. Implement enough of the couchdb backend that reading feeds (and marking feed entries as read) is possible
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 154
diff changeset
94 def _markSelectedFeedEntryRead(self):
153
65c4bb6d5add move management of the selected feed entry into the backend - sqlachemy backend works, couchdb backend currently broken
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 151
diff changeset
95 self.selectedFeedEntry.markRead()
65c4bb6d5add move management of the selected feed entry into the backend - sqlachemy backend works, couchdb backend currently broken
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 151
diff changeset
96
65c4bb6d5add move management of the selected feed entry into the backend - sqlachemy backend works, couchdb backend currently broken
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 151
diff changeset
97 def markFeedEntriesAsRead(self, indices):
65c4bb6d5add move management of the selected feed entry into the backend - sqlachemy backend works, couchdb backend currently broken
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 151
diff changeset
98 for index in indices:
65c4bb6d5add move management of the selected feed entry into the backend - sqlachemy backend works, couchdb backend currently broken
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 151
diff changeset
99 self.feedEntries[index].markRead()
65c4bb6d5add move management of the selected feed entry into the backend - sqlachemy backend works, couchdb backend currently broken
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 151
diff changeset
100 self.session.commit()
65c4bb6d5add move management of the selected feed entry into the backend - sqlachemy backend works, couchdb backend currently broken
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 151
diff changeset
101
65c4bb6d5add move management of the selected feed entry into the backend - sqlachemy backend works, couchdb backend currently broken
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 151
diff changeset
102 def toggleSelectedFeedEntryRead(self):
65c4bb6d5add move management of the selected feed entry into the backend - sqlachemy backend works, couchdb backend currently broken
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 151
diff changeset
103 self.selectedFeedEntry.toggleRead()
65c4bb6d5add move management of the selected feed entry into the backend - sqlachemy backend works, couchdb backend currently broken
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 151
diff changeset
104 self.session.commit()
151
bca9341dc67f move the selected feed into the backend - sqlalchemy backend works, couchdb backend currently broken
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 146
diff changeset
105
128
32a173cb081c move updating the feeds to the backend
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 126
diff changeset
106 def expireFeedEntries(self):
32a173cb081c move updating the feeds to the backend
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 126
diff changeset
107 logger = logging.getLogger("feedupdater")
32a173cb081c move updating the feeds to the backend
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 126
diff changeset
108 expireDate = self._calculateExpireDate()
32a173cb081c move updating the feeds to the backend
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 126
diff changeset
109 logger.info("expiring entries older than " + str(expireDate))
32a173cb081c move updating the feeds to the backend
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 126
diff changeset
110 feedEntry = Mapping.feedEntryTable
32a173cb081c move updating the feeds to the backend
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 126
diff changeset
111 deleteStatement = feedEntry.delete().where(
32a173cb081c move updating the feeds to the backend
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 126
diff changeset
112 and_(feedEntry.c.create_timestamp < expireDate, feedEntry.c.read == 1)
32a173cb081c move updating the feeds to the backend
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 126
diff changeset
113 )
32a173cb081c move updating the feeds to the backend
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 126
diff changeset
114 deleteStatement.execute()
32a173cb081c move updating the feeds to the backend
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 126
diff changeset
115 self.session.commit()