# Topic: Caching # Description: Caching of database-queries on DGS # # Author: Jens-Uwe Gaspar, DGS 'juga' # Document Status: initially created 21-Sep-2012 ## /* ## Dragon Go Server ## Copyright (C) 2001- Erik Ouchterlony, Jens-Uwe Gaspar ## ## This program is free software: you can redistribute it and/or modify ## it under the terms of the GNU Affero General Public License as ## published by the Free Software Foundation, either version 3 of the ## License, or (at your option) any later version. ## ## This program is distributed in the hope that it will be useful, ## but WITHOUT ANY WARRANTY; without even the implied warranty of ## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the ## GNU Affero General Public License for more details. ## ## You should have received a copy of the GNU Affero General Public License ## along with this program. If not, see . ## */ Topics: 1. Introduction 2. Installation 2a. [APC]-cache 2b. [FILE]-cache 3. Configuration 4. Cache Administration 5. Cache Groups 6. Design, Classes and Files (implementation details) 7. Add new cache-group 8. Cache-Group Specifics 9. Other Caching Strategies 9a. Quick-Status 9b. RSS/WAP 9c. Grabbing Web-Output 9d. Same-Thread Caching Appendices: A. Analysis Report B. Future Enhancements #-------- (1) Introduction ---------------------------------------------------- After the latest big upgrade DGS 1.0.15 in June 2012 the server was pretty slow, because of lot of additional database queries (new features or the split of the Players-table to name a few). For the quick-status (QST, see 'specs/quick-suite.txt') and the RSS- & WAP-status page some quota-control and caching had been added before DGS 1.0.15 already to reduce the server-load with a different implementation: # Quick-Status - Progressive Block-Specific Caching # RSS/WAP/Quick-Status see section "(9) Other Caching Strategies" The MAIN "caching" this document is about is describing the CACHE-GROUP-caching (see section (5), (6) and (8)), that caches data that directly resulted from database-queries. A new implementation, especially with a shared memory caching was aimed at as that would have more potential than the previous caching of the alternative-interfaces (RSS/WAP/Quick-Status). The server was still barely usable also because the server machine was pretty old and the server had only 1GB(!) of memory. With the webserver and the database on the same machine running, slow disc-I/O the performance was so bad, that for example sending mails took lowest priority and sending one mail could take up to 50 secs. Therefore an analysis on database-queries was done (see Appendix (A)) to identify queries, that are worth caching to reduce the server-load. The ideal candidates would be queries that only are dependent on a small set of variables, are often loaded and are not updated too frequently. If a data-set is updated and if it's cached, the cache-entry must be either invalidated or the expire-time must be chosen so small, that the invalidation of the cache-entry is not necessary (accepting that showing the updated data in a delayed fashion is acceptable). For each candidate that is cached an expire-time is defined, which should not be longer than one day to keep the number of cache-entries reasonable small. To ensure that, a cache cleanup-script removes cache-entries, that are stored for too long. Some example considerations: - loading the Players-table in 'is_logged_in()'-function can NOT be cached as it changes with every user visit -> therefore no candidate for caching - the games-list on the status-page can be stored in a cache, but only with a small expire-time of 1 min: The status-games show a players last-access-time, which can be updated every second. The "1 min" is short enough as it could be an acceptable time-window to delay showing the updated opponents last-access-time. If for running your DGS-server this is not acceptable it's possible to disable caching for particular cached queries. #-------- (2) Installation ---------------------------------------------------- Important Note: * The DGS server can be used WITHOUT ANY caching! Caching is purely optional, but it is recommended as it can take away load from the database. The cache is currently based on two implementations, which are provided to have the possiblity to adjust the DGS config to the servers restrictions on memory and disc-space. Using the cache-implementation is abstracted in a generic interface 'AbstractCache' to be independent of the implementation. Current cache implementations are: * [APC] shared-memory cache (APC) * [FILE] file-based cache * another older file-based cache, see "(9a) Quick-Status" and "(9b) RSS/WAP" and function 'enforce_min_timeinterval()' These should be replaced in the future using the [FILE]-cache instead, as the latter has a better implementation on file-locking. #-------- (2a) [APC] cache ---------- The [APC]-cache is the preferred cache as cache-entries are stored in memory. It is very fast, but requires to reserve a block of server memory. So this is only possible to use if sufficient memory is available. APC is a PECL extension for PHP, which is not bundled with PHP. The installation is explained on: http://www.php.net/manual/en/book.apc.php Part of the the APC package is the admin-script "apc.php". On the live-server it must be installed at the location "scripts/apc-live.php". The script "apc.php" committed in the DGS-code is exactly that script but with a specific version. You may have a newer version, that should be used and configured accordingly. This script has two ways of configuring for admins. Authorization allows to view the cache-entries. These can contain sensitive information and must therefore be protected against unintended access. The 'scripts'-folder is already protected with a web-password with base-authentification. So the recommended configuration in the apc-script is to set the default "USE_AUTHENTICATION" to "0". If you set an admin-password in the apc-script, take note, that it must be the same as the one the 'scripts'-folder is protected with. Otherwise you are not able to enter the scripts in the 'scripts'-folder as the web-server only keeps track of ONE realm for base-authentification. #-------- (2b) [FILE] cache ---------- The [FILE]-cache only needs enough disc-space and the "DATASTORE_FOLDER"-variable configured in "include/config-local.php" (see 'INSTALL'-file). Take note, that the "DATASTORE_FOLDER" must reside outside of the webservers document-root. Otherwise (ab)users would have access to sensitive data, that is stored in the file-cache. #-------- (3) Configuration --------------------------------------------------- The server-configuration "include/config-local.php" contains 3 important configs to setup the caching for db-queries (this does not affect the caching for the RSS/WAP/quick-status): * DGS_CACHE : default cache-implementation to use - CACHE_TYPE_NONE = caching for db-queries CACHE_GRP_... is disabled - CACHE_TYPE_APC = use [APC] cache-implementation as default - CACHE_TYPE_FILE = use [FILE] cache-implementation as default To use [APC] the APC-package must be installed. To use [FILE] the "DATASTORE_FOLDER" must be set appropriately. Also see section "(2) Installation". * DBG_CACHE : - value >0 writes debugging of caching-statements into error-log - value 0 disables debugging of caching * $DGS_CACHE_GROUPS = array(...) - With this array you can define what cache-type to use for specific cache-groups. If no cache-type is given (e.g. commented out), the default defined by "DGS_CACHE" is used instead. - This setting is only relevant if DGS_CACHE is enabled at all. - When the cache-type is changed, remember to clean/clear the specific cache-group cache-entries (or the full cache) on the (live-)server to save space. - Example: - If the default is CACHE_TYPE_FILE and you have enough shared memory to spare, some cache-groups can be configured to use CACHE_TYPE_APC instead. #-------- (4) Cache Administration -------------------------------------------- There are several scripts to administrate the caching for the db-queries and also for the older RSS/WAP/quick-status caching using the data-store: * 'scripts/index.php' : list with all admin-scripts, see section "Cache Admin" * 'scripts/dgs_cache_admin.php' : Admin page to control db-query caching * "List Cache Group" : lists all cache-groups with their cache-type in use, the cleanup-time (Expire) and some statistics about the number of entries, the consumed size in KBytes, the number of hits (entry read from cache) and misses (entry not found in cache). The "misses" are not available for [APC]-cache, and for the [FILE]-cache only if the APC-cache is available as the [FILE]-cache uses APC to keep track of hits and misses. * "Cleanup All Expired" : Each cache-entry has its own expire-time and each cache-group has its own defined cleanup-expire-time. A "cleanup" removes all cache-entries that are older than the cleanup-expire-time. * "Clear APC Cache" : The [APC]-user-cache is cleared completely. * "Clear File Cache" : The [FILE]-cache for all cache-groups is emptied. These two above are sometimes required after a live-server update, especially if cached database-table have been altered and old cache-entries would contain wrong data-structures. * Cache-Group specific actions: * "Clear" : this action clears the cache only for the specific cache-group * "Clean Expired" : this action cleans up the cache but only for the specific cache-group. For the meaning of "cleanup" see the action above "Cleanup All Expired", but just applied to one cache-group. * 'scripts/apc-live.php' : this script controls the [APC]-cache - shows overview of cache-config, memory usage, cache-entries of user- and opcode-cache * 'scripts/apc_cache_info.php' : this script shows the full meta-content of the [APC]-cache * 'scripts/apc_clear_cache.php' : this script clears the complete APC-cache: all user-cache-entries and all entries from the op-code-cache (which is used to buffer the PHP-files for the web-browser). * 'scripts/clear_datastore.php' : this script allows to empty directories of the file-based caches stored in the "DATASTORE_FOLDER"-directory. * 'hourly_cron.php' : this scripts runs every hour and cleans up old cache-entries to keep the used cache-size reasonable small. #-------- (5) Cache Groups ---------------------------------------------------- A so-called "cache-group" represents a group of cache-entries all created from one database-query with or without dependent variables. All entries for one cache-group must share the same cache-key-prefix, which should also be unique amongst all cache-groups, so that single entries of cache-groups can be deleted easily without disturbing other entries by using a pattern-match on the keys. The PHP-include "include/cache_globals.php" contains the definitions of all the cache-groups in use for the server. If the use is enabled or disabled can be configured; see section "(3) Configuration". The comments behind the different cache-groups give expected size, expire-time and some size-requirements for [APC]-cache, that are likely to be needed for the live-server. The numbers are based on some statistics from the live-server, which are given in the comments above the constants and estimates. #-------- (6) Design, Classes and Files (implementation details) -------------- There are four classes to control the caching of db-queries: DgsCache = a singleton-wrapper to cache objects loaded by database-queries AbstractCache = abstract class with a generic API for cache-implementations ApcCache = implementation for APC-based caching FileCache = implementation for file-based caching Class DgsCache: - basic cache-operations : fetch, store, delete - cache-entries are grouped in so-called "cache-groups" (CACHE_GRP_...), which identify different resultset of one database-query with different dependent variables: see also section "(5) Cache-Groups". For example, there's a cache-group for loading the ConfigPages-table. - Cache-entries for a certain cache-group may be required to be grouped variable-dependent into a meta-group. This meta-group is used to be able to invalidate all cache-entries that belong to only one of a set of dependent variables. For example, storing cache-entries for observers of a game is dependent on the game-id (gid) and a user-id (uid). The use-cases for invalidation of the cache-entry knows only the game-id. This problem is solved by specifying a meta-group, that collects all such cache-entries. # cache-group CACHE_GRP_GAME_OBSERVERS for game-id 12345 cache-key for user 4711: "Observers.12345.4711" cache-key for user 4712: "Observers.12345.4712" # meta-group (user-id independent) cache-group-key: "Observers.12345" So storing a single cache-entry is done with: DgsCache::store( .., CACHE_GRP_GAME_OBSERVERS, "Observers.12345.4711", .., .., "Observers.12345" ); DgsCache::store( .., CACHE_GRP_GAME_OBSERVERS, "Observers.12345.4712", .., .., "Observers.12345" ); Invalidating is then either possible for a single cache-key or for the cache-group-key deleting all collected single entries. For such a meta-group-key a separate cache-entry is stored with an array to reference all single cache-keys. - Sometimes a different behaviour is required in case the caching is disabled, or a cache is not persistent over different web-requests. To act on this, there are functions to determine if the caching is disabled or if the caching is persistent. For example this is currently used for caching the Clock- and ConfigPages-table. See also section "(8) Cache-Group Specifics" Class ApcCache / FileCache: - These classes contain not only the implementation for the AbstractCache-functions, but also function that are directly used for the cache cleanup- and admin-scripts. - For the FileCache a statistics about MISSes and HITs of the cache-entries is created and stored in the [APC]-cache (only if it's supported of course). The stored data only needs ca. 2 KB, that can be spared from regular caching- entries. The number of misses & hits are shown on the 'scripts/dgs_cache_admin.php'-script. #-------- (7) Add new cache-group --------------------------------------------- Important Notes: * Before you make a new cache-group be aware of possible race-conditions, that can occur because of cached-data. So it is not advisable to load cached data if updates are done dependent on the cached data! To add a new cache-group the following steps are required: * Adjust "include/cache_globals.php": - Add a new "CACHE_GRP_..." definition (use consecutive numbers) - Adjust the "MAX_CACHE_GRP" definition in there accordingly - Add a new line in the array "$ARR_CACHE_GROUP_NAMES" for the new cache-group specifying the prefix for the cache-key to be used. This is used to be able to find cache-entries in the caches to clear or cleanup those entries. - Add a new line in the array "$ARR_CACHE_GROUP_CLEANUP" specifying a cleanup-time (=expire-time for cleanup) for the new cache-group. This expire-time must be greater than or equal to the highest expire-time you use to store a cache-entry for the new cache-group. If you think no cleanup is required (for example for cache-entries that are independent of any variables), then the entry can be commented out. For example see the CACHE_GRP_CLOCKS-entry. * Adjust "include/config.php": - Add a line in the array "$DGS_CACHE_GROUPS" for the new cache-group with a cache-type that makes sense, but comment it out. For a specific server-installation the new config have to be added in the servers local configuration "include/config-local.php" as well. * Adjust "include/config-local.php" on the live-server: - Configure the new cache-group according to the specific needs of the live-server. * Adjust "scripts/updates//other_changes.txt": - Add a new configuration also in the "scripts/updates//other_changes.txt"-file, so that it is not forgotten on server-upgrades. * Use DgsCache::fetch/store/delete to fetch, store and invalidate (=delete) cache-entries. Think for which use-cases invalidation is needed, if you need a meta-group or not. Think WHAT you want to store in the cache-entry. You should not store the database-result directly as no mysql-binding can be cached (or should be). Also think about in which format it is stored. The more complex a data structure is, the more space is required in the cache. Especially for the [APC]-cache a single object can take a LOT of space. You can use the apc-script to view a single entry and the size-requirement. A thumb of rule is, that the space required in the APC-cache is 5 times larger than a serialized data object in the [FILE]-cache would need. The simplest data structure to store often is a plain array. You should NOT cache PHP-objects, that contain redundant data as that eats up additional space in the cache. For example, the bulletins-view for a user on the web-status-page is cached (see cache-group CACHE_GRP_BULLETINS). After loading the data from the database, the normally resulting data consists of a list of Bulletin-objects with embedded original raw data from the database. So for this example, the original raw-data is cached instead of the Bulletin-objects (with the raw-data). Then on loading the data from the cache, the Bulletin-objects are re-assembled into the expected list of Bulletins; see function "Bulletin::load_cache_bulletins()". * Adjust section "(8) Cache-Group Specifics" in "specs/caching.txt" if there's something special about the new cache-group, that is worth mentioning for other developers. #-------- (8) Cache-Group Specifics ------------------------------------------- This section describes each cache-group (sometimes summoned) with a motivation for caching, a description, for which use-cases a cache-invalidation is needed, the expire-time and if needed what special handling is used. See "template" for new cache-group at end of this section. * CACHE_GRP_CFG_PAGES - Before DGS 1.0.15 (Jun-2012), all fields from the 'ConfigPages'-table were in the Players-table, which is always loaded. The splitted away ConfigPages- table contain fields to configure the table-columns for table-lists and some configs for the status- and forum-pages. So whenever these pages are loaded, the ConfigPages-entry for the user must be loaded as well. The split of the Players-table was done, because it grew a bit too big with all the new features to make up a good table-design. The configuration stored in the ConfigPages-table is not changed so often, so it made sense to store it in the cache. - The previous code was using "ConfigPages::load_config_pages()" which only loaded the one specific table-field which is needed for a loaded page. When the caching is disabled for this cache-group, this code is still used to avoid to load too much data at once when it's not needed. However, when the cache is used ALL fields needs to be loaded. This requires to check, if the caching is persistent (with different threads). If it is persistent, all fields must be loaded and stored in the cache, while otherwise only the one field required need to be loaded. - All required cache-functions for this cache-group can be found in the 'ConfigPages'-class. The function to load db-data and/or to fetch it from the cache is "ConfigPages::load_config_pages". - The cache-entry is dependent only on the user-id [=Players.ID] - A cache-entry is invalidated when the table-columns are hidden/shown, the folders to show on the status-page are changed, or when the forum-thread- view options have changed. All these actions are considered to happen sparsely enough to justify an expire-time of 1 day. * CACHE_GRP_CFG_BOARD - Before DGS 1.0.15 (Jun-2012), all fields from the 'ConfigBoard'-table were in the Players-table, which is always loaded. The splitted away ConfigBoard- table contain fields to configure the GUI of the game-page. The configuration stored in the ConfigBoard-table is not changed so often. - The cache-functions for this cache-group are in the 'ConfigBoard'-class. - A cache-entry is dependent only on the user-id [=Players.ID] - A cache-entry is invalidated when the preferences for board-GUI in the user-profiles are changed. This update is assumed to not happen often, therefore using an expire-time of 1 day. * CACHE_GRP_CLOCKS - Before DGS 1.0.15 (Jun-2012), there was the 'ClockCache'-class (see also section "(9d) Same Thread Caching"), which was used to cache Clock-data within one thread for one web-request to prevent loading the same clock-entry multiple-times as the code is quite fragmented and different portions need the clock-data. The cache-entries are load from the 'Clocks'-table. This caching was enhanced by caching ALL (currently 56) Clock-entries, which are not that many right now and will not increase by much in the future. Even if the Clock-ticks are updated every 5 minutes by the clock-tick-cron- script, the Clock-data is used very often, which justified putting it into a persistent cache with an expire-time of 10 minutes. - When the caching is disabled for this cache-group, the one-thread- caching is still used loading only one specific Clock.ID at a time to avoid to load too much data at once when it's not needed. However, when the cache is used ALL Clocks need to be loaded. This requires to check, if the caching is persistent and if it is, all clocks must be loaded and stored in the cache. - All required cache-functions for this cache-group can be found in the 'CacheClock'-class. The function to load db-data and/or to fetch it from the cache is "ClockCache::load_clock_ticks()". - The cache-entry is invalidated when the Clocks are updated in the "clock_ticks.php"-script. * CACHE_GRP_GAME_OBSERVERS - A game can be observed by the players or viewers of a running game. The data is loaded from the 'Observers'-table. There are two calls: - 'is_on_observe_list(gid,uid)' to check if a user is observing a game - 'has_observers(gid)' to check if there are any other observers As currently (DGS 1.0.15) the game is frequently loaded to just browse the move-history of a game by the same users within a shorter amount of time, both of these information about observers is combined and stored as cache-entries using the function "check_for_observers()". - Because one of the source-functions depends on both game-id and user-id while the other only depends on game-id, a meta-group is needed for each game-id, that collect each cache-entry for the same game but by different users. Cache-invalidation is needed if any users changes the observer-status on a game (game-id). Additionally the cache-entries are deleted, if all observers of a game are deleted (e.g. when the game is deleted or is finished). - As the caching is only meant to decrease server-load for the case of browsing the move-history of a game, the expire-time for the cache is only 30 minutes. * CACHE_GRP_GAME_NOTES - A game can have private game notes by the players stored in the 'GamesNotes'-table. Currently (DGS 1.0.15) the game is frequently loaded to just browse the move-history of a game by the same users within a shorter amount of time, also the game-notes need to be loaded. - Cache-invalidation is needed if a player changes the game-notes, so they are dependent on game-id and user-id. - As the caching is only meant to decrease server-load for the case of browsing the move-history of a game, the expire-time for the cache is only 30 minutes. * CACHE_GRP_USER_REF - The user-reference tag is used in forum-pages or private messages in the form or . The output shows the Players.Handle and Players.Name. - A cache-entry for this cache-group is dependent either of the user-id or by user-handle, so on invalidation (update of handle or name), both potential entries must be deleted from the cache. Problem with the user-handle is, that it is a text, so the user can use arbitrary case for the letters in the handle. Therefore the case of the cache-key is converted into lower-case to be able to identify it on invalidation. - The user-handle can only be changed by an admin. The user-name can be changed by the player or an admin at any time. However, that is rarely done for most of the users, so an expire-time for the caching-entries of 1 day was chosen. The cleanup-expire time is even greater as it's really rare, that the handle or name is changed. * CACHE_GRP_STATS_GAMES - The statistics-page accumulates the number and sum of moves for all game status values. This db-operation can take quite a while, because it has to lock the Games-table, which is a frequently used db-table. Therefore the cumulated values are cached. - Because there are two sets of cumulated values, a meta-group is used to make it easy to delete all relevant cache-entries. At the moment there are only 2 cache-entries not depending on further variables. In the future more cached-entries are thinkable. - Cache-invalidation is only done on a daily-basis by the daily-cron-script, that also generates entries in the Statistics-table, though there is no direct connection to this cache-group the invalidation takes place in that script. The cumulated number/sum for game-moves could be interesting for a shorter time-interval than one day, but as the data-creation can be quite expensive, the longer expire-time of 1 day was considered acceptable. - There is another caching in use to store the generated graphics. For this description see section "(9c) Grabbing Web-Output". * CACHE_GRP_FOLDERS - The 'Folders'-table contain customized message-folders for the users. Folder-information is used in various places (message-lists, message-search, status-page, message-thread) and luckily does not change often, so being a good candidate for caching. - When the caching is disabled for this cache-group or a non-persistent- cache-implementation would be used, only specific folders can be loaded. For caching the full set for a user is loaded and cached. - A cache-entry only depends on the user-id. - Cache-invalidation is only needed when some data on the user-folders is changed. - Expire-time for the message-folders is 1 day. * CACHE_GRP_PROFILE - The 'Profiles'-table is used to store search-profiles for pages with table-lists and store templates. The search-profile for a page with table-list is always loaded, so this is a candidate for caching to reduce the server-load. - A cache-entry depends on user-id and the profile-type (page-dependent). - The cache-entry for user-id and search-profile of a table is invalidated, if that specific search-profile is saved or deleted. - The expire-time for a search-profile is 1 day as it's not expected to change frequently. - All required cache-functions for this cache-group can be found in the 'Profile'-class. * CACHE_GRP_GAME_MOVES * CACHE_GRP_GAME_MOVEMSG - Currently (DGS 1.0.15) the game is frequently loaded to just browse the move-history of a game by the same users within a shorter amount of time. This makes the list of moves and corresponding move-messages loaded from the 'Moves'-table (and 'MoveMessages'-table respectively) a good candidate for caching. - The moves (and messages) are only cached when the caching is using a persistent cache and only for viewing a game. Caching is never used to pre-load data to handle some processing (like making a move for example) as caching would be a great danger of race-conditions or inaccurate data, which could be easily lead to inconsistencies. - A cache-entry depends only on the game-id. - Cache-entries for cache-group CACHE_GRP_GAME_MOVES are invalidated on every occasion someone has to insert an entry into the 'Moves'-table (e.g. moving, passing, scoring, adding-time, creating or deleting a game; the latter two cases need no invalidation, but it doesn't hurt either). - Cache-entries for cache-group CACHE_GRP_GAME_MOVEMSG are invalidated on every occasion someone has to insert an entry into the 'MoveMessages'-table (e.g. move-actions with a comment). - Storing the moves can consume quite some space, therefore only a short time of 10 mins is chosen as expire-time, or even less with 5 mins for finished games or game in fair-komi-negotiation. * CACHE_GRP_COND_MOVES - The game-page is frequently loaded to just browse the move-history of a game by the same users within a shorter amount of time. To reduce the db-load caching the conditional-moves that might exist for the players are cached. For observers there are no conditional-moves, that need to be loaded or cached. - A cache-entry depends on the game-id and user-id, but for game-deletions also a group-cache-entry is stored to invalidate cached conditional-moves entries for both players. - Cache-entries for this cache-group are invalidated on every edit or moves played from the conditional-moves-sequence, i.e. an update on the corresponding 'MoveSequence'-table. - Expire-time of 1 hour is used. * CACHE_GRP_TOURNAMENT - Once a tournament has been created, there are normally not longer changes to the Tournament-table except for registration of users. There are some changes by the tournament-system, but the 'Tournament'-table is loaded very often for all the tournament-features, so caching is almost inevitable. - A cache-entry depends only on the tournament-ID (tid). - There is the "TournamentCache"-class, which is used for all kind of caching for tournaments. For locking the tournamnet by the cron-script, an additional soft-caching kicks in if the caching for this cache-group is not persistent or disabled. For this also see function 'TournamentCache::release_tournament_cron_lock()'. - Cache-invalidation is done on any change of the 'Tournament'-table. See also "Tournament"-class. Entries are also deleted for releasing the locking by the cron-script as mentioned above. - Expire-time of 1 hour is used, because of the moderate volatility mentioned above. * CACHE_GRP_TDIRECTOR - 'TournamentDirector'-table - expire 1d - comment not cached, only if TD + flags - The tournament-directors are not changed that often once a tournament has been created and started, but are used a lot for tournament-features. Especially a check is done, if a user is a tournamnent-director or not, and which admin-roles (saved in the Flags-field) he/she is able to fulfill. Though for example the Comment-field is only needed for the director-list. - A cache-entry is storing all entries with user-id + flags for a specific tournament-ID (tid) from the 'TournamentDirector'-table, which is the only dependency. - The "TournamentCache"-class is used as wrapper to load cached data for these db-tables, esp. see function 'TournamentCache::is_cache_tournament_director()'. - Cache-invalidation is done on adding, changing or deleting a director of a tournament. - Expire-time of each cache-entry is 1 day. * CACHE_GRP_TLPROPS (caching of the 'TournamentLadderProps'-table) * CACHE_GRP_TPROPS (caching of the 'TournamentProperties'-table) * CACHE_GRP_TRULES (caching of the 'TournamentRules'-table) * CACHE_GRP_TPOINTS (caching of the 'TournamentPoints'-table) - Once a tournament has been created and started, there are normally not longer changes to these db-tables except for admin-intervention. Also these tables are often used to configure tournaments on various pages, so their content is often required for a specific tournament. - A cache-entry for these tables depends only on the tournament-ID (tid). - The "TournamentCache"-class is used as wrapper to load cached data for these db-tables. - Cache-invalidation is done on any change of the respective db-table (except for an insert as that is only used for tournament-creation). - Expire-time of each cache-entry is 1 day. * CACHE_GRP_TROUND - For round-robin-tournaments, rounds (stored in the 'TournamentRound'-table) can be added, that contain configurations of tournament-pools. This data does only change during setup of a tournament. So caching the round-data makes sense as the data is often needed to just view tournament-data. - A cache-entry for these tables depends on the tournament-ID (tid) and round. - The "TournamentCache"-class is used as wrapper to load cached data for these db-tables. - Cache-invalidation is done on any change of the 'TournamentRound'-table. - Expire-time of each cache-entry is 1 day. * CACHE_GRP_TNEWS - Tournament-News are shown on the view-tournament-page, which is presumably often used to get a status of the tournament. Tournament-News, once entered, do not change often during a day. Therefore they are cached to reduce general server load. - A cache-entry for these tables depends only on the tournament-ID (tid). - The "TournamentCache"-class is used as wrapper to load cached data for these db-tables. - Cache-invalidation is done on any change of the 'TournamentNews'-table. - Expire-time of each cache-entry is 1 day. * CACHE_GRP_TP_COUNT (caching of the 'TournamentParticipant'-table) * CACHE_GRP_TPARTICIPANT (caching of the 'TournamentParticipant'-table) - The number of registered participants of a tournament is needed to be shown on viewing the tournament-info-page. Also wether a user is already registered in a tournament is often checked as well as the tournament-rating of a user is needed. Therefore the count of tournament-participants per registration-status is cached as well as the data for single tournament-participants. - A cache-entry for the count depends only on the tournament-ID (tid). - A cache-entry for the tournament-participant depends on the tournament-ID (tid) and the user-id. - The "TournamentCache"-class is used as wrapper to load cached data for these db-tables. - Cache-invalidation for both cache-groups is done on any change of the 'TournamentParticipant'-table. - Expire-time of each cache-entry is 1 hour. * CACHE_GRP_TRESULT - 'TournamentResult'-table - Tournament-Results are shown on the view-tournament-page, which is presumably often used to get a status of the tournament. Therefore they are cached to reduce general server load. - A cache-entry for 'TournamentResult'-table depends only on the tournament-ID (tid). - The "TournamentCache"-class is used as wrapper to load cached data for these db-tables. - Cache-invalidation is done on any change of the 'TournamentResult'-table. - Expire-time of cache-entry is 1 day. * CACHE_GRP_GAMES - As currently (DGS 1.0.15) the game is frequently loaded to just browse the move-history of a game by the same users within a shorter amount of time. To further reduce the server-load for this case, the 'Games'-table is cached for the game-page. - A cache-entry depends only the game-id. - Cache-entries are invalidated on every change of the 'Games'-table, especially for adding time, deleting game, save komi for fair-komi- negotiation, finishing a game, move-like operations, fix-script for time-left (Games.TimeOutDate), removing a user from a ladder-tournament. - Expire-time of cache-entry is 10 mins. * CACHE_GRP_FORUM - For the forums-thread-list and forum-thread-posts-view the data of the specific forum is required and be cached easily as the relevant and cached forum-data from the 'Forums'-table (name, description, options) does not change often. - A cache-entry for this cache-group depends only on the forum-ID (fid). - Cache-invalidation is done on any change of the 'Forums'-table including forum-administration (e.g. changing flags, name or description), on posting or moderating forum-messages, recalculating and forum-consistency. - Expire-time of of cache-entry is 1 day. * CACHE_GRP_FORUM_NAMES - For the forum-search the list of forum-IDs with names from the 'Forums'-table is cached. - The cache-entry has no dependencies. - Cache-invalidation is done whenever an admin changes the forum-names or their order in the 'Forums'-table. - Expire-time of of cache-entry is 1 day. * CACHE_GRP_GAMELIST_STATUS - Evaluated from some web-log-statistics there are several users, that open the web-status-page quite often even within one minute. To overcome this caching for the status-page game-list (read from the 'Games'-table) has been added. - However, the status- list contains also the opponents last-access-date/time in form of a icon showing when he/she was last "online" and that can change any second; but the used precision for a timestamp is 1 min, so that is used as expire-time for this cache-group. - A cache-entry depends on the user-ID (uid). - Cache-invalidation is done when some field of the status-list is changed, except for the opponents last-access. So cache-entry deletion is done for example for: moving in a game of the old and new player to move next, on fair-komi-negotiation, on updating the next-game-order on the status-page, on creating a game by joining a waiting-room-offer, starting an invitation-game, on deleting and finishing a game, on adding time - Expire-time of each is 1 minute. - (*) IMPORTANT TO NOTE is, that there are some use-cases, that do not invalidate the cache-entry for the users even so they have influence on the status-content. These use-cases are: - starting/ending vacation - the clock-tick-script changing the time for games - game time-left fixing script - change of night-time for users - change of players name, rating and last-access With the short expire-time of 1 minute all the above exceptions to cache- invalidation are accepted to show up delayed (by 1 min). - Currently (Nov-2012) this cache-group is disabled on the live-server. Mainly out of reasons mentioned above at (*). However, if you have your own server, you could accept to enable caching for this cache-group if 1 min delay is acceptable for the use-cases mentioned. * CACHE_GRP_BULLETINS - Bulletins shown on the web-status-page do not change often over the day. The status-page is one of the pages that is requested a LOT, so every reduction of additional db-query that can be avoided to reload is favourable to server-load and to response-time for the users. Related db-tables are: 'Bulletin', 'BulletinRead', 'BulletinTarget' - The 1st three bulletins are shown if there are some new or changed ones. That is decided upon a global lastchange-date for all bulletins. This date is cached in a single cache-entry within this cache-group. Besides that the read-state for bulletins on the status-page is dependent on the user-ID (uid). - Cache-invalidation of the user-dependent cache-entries is done on any change of a players bulletin-new-count, which is updated when the user opens a web-page showing the status-menu; and therefore indirectly triggered after every update on bulletins. Also when the user marks a bulletin as read the cache-entry is deleted. The single global cache-entry is only updated (not deleted normally). - Expire-time of cache-entry is 1 day. * CACHE_GRP_MSGLIST - Private messages read from the 'Messages'-table that are new or needs a reply are normally shown on the web-status-page (can be configured though). The status-page is one of the pages that is requested a LOT, so every reduction of additional db-query that can be avoided to reload is favourable to server-load and to response-time for the users. - A cache-entry for this cache-group depends only on the user-ID (uid). - Cache-invalidation is done on sending a private message, editing folders or moving the message into different folders. - Expire-time of cache-entry is 4 hours (to avoid the needed cache-size grows too big). * CACHE_GRP_TGAMES - Each tournament-type has a page to view the current games for the players. Right now there is a ladder-view for ladder-tournaments and a pool-view for round-robin-tournaments. With BIG tournaments the data to be loaded can be quite big as well. The full ladder or all pools would need to be loaded for every user. For larger tournaments this could easily be around 1000-2000 rows. Therefore it's imperative to try to reduce to reload the same data over and over if it's not changing too often. - The 'TournamentGames'-table contain the started and running game-IDs for a tournament along with the results of already finished games. Therefore the rows from this table are cached. - A cache-entry for this cache-group depends on the tournament-ID (tid), the round-ID and the pools (the round-ID and pool-info is static for ladder- tournaments). For invalidation the specific round or pool-info is not available, so a meta-group is used to collect all cached entries for a specific tournament. - Cache-invalidation for this cache-group is done on tournament game-administration, starting a ladder-challenge, on updating the game-wait-state, starting a tournament-round-game or on finishing a tournament-game. - The "TournamentCache"-class is used as wrapper to load cached data for these db-tables. - Expire-time of cache-entry is 1 day. * CACHE_GRP_TLADDER - Right now there is a ladder-view for ladder-tournaments. With BIG tournaments the data to be loaded from the 'TournamentLadder'-table can be quite big as well. The full ladder would need to be loaded for every user. For larger tournaments this could easily be around several hundred rows. Therefore it's imperative to try to reduce to reload the same data over and over if it's not changing too often. Also on the tournament-info for ladder-tournament the top 5 ladder-standings are shown, which is loaded from the same table. So some caching can be combined here. - A cache-entry for these tables depends on the tournament-ID (tid), the limit first asked and a boolean state telling if the tournament-rating is included in the ladder-view or not. Regarding the "limit": if the ladder-standings are first asked then not the whole ladder is loaded. If on the other hand the full ladder is loaded, then for the ladder-standings the full ladder is used but shortened to the 5 entries shown. On the various occasions where a cache-invalidation is needed, not all of the dependent variables are available. So a meta-group is used to collect all related cache-entries for a specific tournament-ID. - Cache-invalidation for this cache-group is done on editing the ladder-properties, starting a new ladder-challenge, changing the ladder-rank of a user, adding a user to the ladder, on deleting and seeding the ladder and after processing the game-end for the tournament. - The "TournamentCache"-class is used as wrapper to load cached data for these db-tables. - Expire-time of cache-entry is 1 hour. * CACHE_GRP_TPOOLS - Right now there is a pool-view for round-robin-tournaments. With BIG tournaments the data to be loaded from the 'TournamentPool'-table can be quite big as well. The full pool-view would need to be loaded for every user. For larger tournaments this could easily be around several hundred rows. Therefore it's imperative to try to reduce to reload the same data over and over if it's not changing too often. - A cache-entry for these tables depends on the tournament-ID (tid), tournament- round (Round) and if the tournament-rating is included. On the various occasions where a cache-invalidation is needed, not all of the dependent variables are available. So a meta-group is used to collect all related cache-entries for a specific tournament-ID and tournament-round. - Cache-invalidation for this cache-group is done on creating and seeding pools, editing the pool-rank of players, filling ranks, setting pool-winners. All of these operations are executed manually by a tournament-director. Processing the game-end for the tournament does not change the pool-data, but only the tournament-games, and therefore needs no invalidation. - The "TournamentCache"-class is used as wrapper to load cached data for these db-tables. - Expire-time of cache-entry is 1 hour. * CACHE_GRP_GAMESGF_COUNT - A game can have attached SGFs stored in the 'GamesNotes'-table. If there are attached SGFs is indicated by a set 'AttachedSgf'-games-flag. Currently (DGS 1.0.15) the game is frequently loaded to just browse the move-history of a game by the same users within a shorter amount of time, also the count of attached SGFs needs to be loaded. - Cache-invalidation is needed if the Games.Flags 'AttachedSgf' is changed, so there's only a dependency on game-id. - As the caching is only meant to decrease server-load for the case of browsing the move-history of a game, the expire-time for a cache-entry is only 30 minutes. * CACHE_GRP_USER_HANDLE - Caching of data for a user-handle is used for previewing rating-changes. - A cache-entry for this cache-group depends on the user-handle. The cache-key (user-handle) is converted into lower-case to be able to identify it independent from the writing. - Only automatic expire-invalidation is performed as the rating does not change so often. Current expire-time of cache-entry is 1 hour. * CACHE_GRP_TVISIT - Cache visits (="viewing") of tournament-info-page by users, which is used to calculate count of new tournaments to join, that users have not seen yet. - A cache-entry for these tables depends on the user-id (uid) and contains a list of tournament-ID (tid) that have been visited by a user. - Cache invalidation is done for a user, if a visit has been added. - The "TournamentCache"-class is used as wrapper to load cached data for these db-tables. - Expire-time of cache-entry is 7 days. * CACHE_GRP_CONT_FP_HIDE - Viewing forum-posts on the forum-read and forum-search page should use a cache to load the forum-post hide-list of a user from the Contacts-table. - A cache-entry depends on the users contact-list with the forum-post-hide system-flag being set. Cache can currently only be used for one specific system-flag. - Cache invalidation is done for a user, if contacts are added or updated. - The "Contact"-class is used as wrapper to load and store this cached data. - Expire-time of cache-entry is 3 days. * CACHE_GRP_RATINGLOG - The 'Ratinglog'-table is used to store rating-changes for users. As speed-up on creating the rating-graph some of the ratinglog-fields for a user will be cached for that. - A cache-entry depends on the user-id. - The cache-entry for user-id is invalidated, if a rated game ends and the rating- changes are written into the Ratinglog-table for both players. - The expire-time for a search-profile is 15 minutes to not blow the cache as the ratinglog-data can be quite large. - All required cache-functions for this cache-group can be found in the 'Ratinglog'-class. * CACHE_GRP_TAGS - The 'Tags'-table contain private customized tags for the users. Tags can be set on forum posts and likely not changed often for a user. - For caching the full tag set for a user is loaded and cached. - A cache-entry only depends on the user-id. - Cache-invalidation is only needed when some data on the tags is changed. - Expire-time for the message-folders is 1 day. * new cache-group CACHE_GRP_ ... - motivation, description, cache-invalidation, expire-time, special-handling ? #-------- (9) Other Caching Strategies ---------------------------------------- Besides the cache-group-based caching mainly described in this document, this chapter lists the other used caching strategies currently (Dec-2012) in use. #-------- (9a) Quick-Status ---------------- In early 2012 (ca. Jan) it was detected, that the RSS- and quick-status-page were excessively used and tremendously slowed down the server due to heavy server- load. Later this was documented on the live-server forums mostly regarding RSS, but also hinting towards quick-status: # 02-May-2012: Too many server requests and countermeasures see http://www.dragongoserver.net/forum/read.php?forum=4&thread=32273 So for the quick-status some form of caching was introduced, which ended in a "Progressive Block-Specific Caching". This form of caching is described in 'specs/quick-suite.txt' section (4.QST) "Important Notes". The quick-status-page uses another kind of file-based-caching, that is not explained in this document, see also function 'enforce_min_timeinterval()'. For consistency-reasons, it would be good to replace this kind of caching with something using the [FILE]- or [APC]-cache. #-------- (9b) RSS/WAP --------------------- In early 2012 (ca. Jan) it was detected, that the RSS- and quick-status-page were excessively used and tremendously slowed down the server due to heavy server- load. For RSS alone 11 Mio page-requests (of 29 Mio, ca. 38%) per month were done. Sometimes even every 5-10 secs the RSS-requests came in (resulting in over 3 Mio request/month): # 02-May-2012: Too many server requests and countermeasures see http://www.dragongoserver.net/forum/read.php?forum=4&thread=32273 This was way too much, so the RSS-pages were restricted with a temporary lock if asked more frequent than 5 minutes. To measure the time since the last query the same kind of caching was used as for the quick-status-page (see section "(9a)") with the difference, that not much content is stored in the cache-files. This "hard" way of blocking requests has been chosen, because the users using RSS were often users, that played not via the web-page but with a mobile device, didn't read private messages or had no email on file that would have allowed some kind of polite request to reduce the queries. Often from studying old webserver-logs and the db-Errorlog it happened, that there were users that left the DGS-site without deactivating the RSS-requests. In summary, the chance to reach those RSS-"abusers" directly were slim. # see also http://www.dragongoserver.net/forum/read.php?forum=4&thread=32273#32280 Therefore a more direct "punch-in-the-face"-approach was chosen blocking off requests, that took place too often giving out a warning (and no content). The "temporary block" is as the word says only temporary. So if the user does not send another RSS-request within 5 mins, the expected RSS-status data is returned. After that measure, the number of RSS-page-requests dropped to ca. 350.000 per month. #-------- (9c) Grabbing Web-Output --------- Since Dec-2008 the graphics that are generated by "statisticspng.php" are saved in a file-based cache-file in the "CACHE_FOLDER"-directory, which is located in the document-root and therefore is publically accessible to the users. The graphic is generated on the server with the GD-library and sent to the requesting user. This bears a disadvantage, because a logged-in users language profile is used to build the texts, that are included in the graphs. To overcome that, there is a 'no_cache'-option. By (manually) setting it to '1' in the URL to the "statistics.php"-page, the cache-files can be invalidated. When stored in the cache the generated image is captured with the function 'grab_output_start()' and 'grab_output_end()' using the ob-handlers from PHP. When restored from the cache the function 'image_passthru()' is used to return the graph-image loaded from the cache-file (if existing). #-------- (9d) Same-Thread Caching --------- If the caching is not using a persistent cache-implementation, there are classes that serve as local cache storing data for ONE web-request. Other web-requests can not see that data, but even within one web-request the same data can be needed several times and to complexity and dependencies of the code, the data can not be passed on via function-calls but need to temporarily stored in a cache to reduce redundant db-queries. * class TournamentCache - The "TournamentCache"-class is used for all kind of caching for tournaments. For locking the tournament by the tournament-cron-script, an additional soft-caching kicks in if the caching is not persistent or disabled. For this also - see function 'TournamentCache::release_tournament_cron_lock()'. - see CACHE_GRP_TOURNAMENT * class ClockCache - The 'ClockCache'-class is used to cache Clock-data within one thread for one web-request to prevent loading the same clock-entry multiple-times as the code is quite fragmented and different portions need the clock-data. #-------- (A) Analysis Report ------------------------------------------------- # candidates for caching: file + TTL + DELCHK (=need-active-delete) + DEP(depend on WHAT object) + estimated size needed # current live-server (Sep-2012) needs more RAM (only 1GB) # Run analysis, sources: # - db_query-log from webserver-error-log (after enabling db_query-logging) on LIVE-server for 8h (with ca. 500.000 db_query-requests) # - web-stats: http://www.dragongoserver.net/stat/ -> TOP 30 URL's # - source-code # check all occurences of the following func-calls # RAW: mysql_query # FUNC: db_query # FUNC: mysql_exists_row # FUNC: mysql_found_rows # FUNC: mysql_single_fetch # FUNC: mysql_single_col # check for the following commands at places to add caching # CHECK: mysql_affected_rows # CHECK: mysql_close # CHECK: mysql_data_seek # CHECK: mysql_error # CHECK: mysql_fetch_array # CHECK: mysql_fetch_assoc # CHECK: mysql_fetch_row # CHECK: mysql_free_result # CHECK: mysql_num_rows #-------------------------------------------------------------------------- # GOOD CANDIDATES: # - First analysis to determine good candidates for caching # - candidate is better if data does not change much #---- funcs: 1w possible, but need to release cache for other entries # HIGH prio ContactPages::load_config_pages.find(uid) 1d DEL-on-upd-user 200B/user => YES, 1d, DEL-on-upd-table(uid); special cache (loading all fields of entry) ConfigBoard::load_config_board.find(uid) 1w DEL-on-upd-user 20B/user => YES, 1d, DEL-on-upd-table(uid) is_on_observe_list(gid,uid) 1d DEL-on-upd-obs 10B/e => YES, 30min, DEL-on-toggle(gid,uid) has_observers(gid) 1d DEL-on-upd-obs 10B/e => YES, 1d, DEL-on-toggle(gid) load_clock_ticks.find(clock_id) 5min DEL-on-run-cron 30B/e * 55 => YES, 5min, DEL-on-cron-clocktick(all); special cache (loading all Clocks) board.load_from_db.find_moves(gid) 1h DEL-on-move 3KB/gid => YES, 1h, DEL-on-move(gid) board.load_from_db.movemessage(gid,MoveNr) 1h DEL-on-move 1KB/e => YES, 1h, DEL-on-move(gid); special cache (loading all move-msgs), needed all? game.gamenotes.single_fetch(gid,uid) 1h DEL-on-upd-user 50B/e => YES, 1h, DEL-on-upd-notes(gid,uid) statistics.games.count_moves(-) 1h DEL-on-move 100B => YES, 1d, EXPIRE statistics.q2(-) 1h DEL-on-move 20B => refactor using "WITH ROLLUP" on statistics.games.count_moves statistics.q3(-) 1h DEL-on-upd-allusers 20B => YES, 1d, EXPIRE # MED PRIO Forum::load_forum2(forum_id) 1w DEL-on-upd-by-adm 200B/e => YES, 1h, DEL-on-fpost(fid); -- load single Forums-entry for title/opts status/quick_status/rss.find_games(uid) 30min DEL-on-move 200B/game*user => YES, 30min, DEL-on-game-act(uids); though MUCH-C-data get_folders(uid) 1d DEL-on-upd-user 100B/user => YES, 1d, DEL-on-upd-folder(uid) Profile::load_profile(uid,Type) 1w DEL-on-upd-user 200B/user => YES, 1d, DEL-on-upd-table(uid) links.load_links(-) 1d DEL-on-upd-ed/tr 3KB => YES, 1d, DEL-on-upd-editor/translator status/message_list_query(uid) 30min DEL-on-msg 50B/msg*user => YES, 30min, DEL-on-msg(uids) # LESS PRIO user_reference.single_fetch(uid) 1d DEL-on-upd-user 20B/e => YES, 1d, DEL-on-upd-name(uid) faq.find_titles(-) 1d DEL-on-upd-ed/tr 60KB => YES, 1d, DEL-on-upd-editor/translator intro.load_intro(-) 1d DEL-on-upd-ed/tr 20KB => YES, 1d, DEL-on-upd-editor/translator status.find_mp_games(uid) 1d DEL-on-user/join 200B/MPgame*user display_posts_pending_approval.find(-) 1h DEL-on-post-pend 1KB => YES, 1h, DEL-on-fpostpend() Bulletin.load_bulletins(uid) 1d DEL-on-new/edit-bull 300B/bull*user => YES, 1d, DEL-on-edit/new-bull(*) translate.groups_query(-) 1d DEL-on-upd-adm 200B => YES, 1d, DEL-on-transl-adm(*) ratingpng.ratingdata(uid) 1h DEL-on-gend 10-50KB/user => YES, 1h, DEL-on-gend(uid) ratingpng.find_max_time(uid) 1h DEL-on-gend 10B/user => YES, 1h, DEL-on-gend(uid) ratingpng.find_initial(uid) 1h DEL-on-gend 50B/user => YES, 1h, DEL-on-gend(uid) Tournament.load_tournament(tid) 1d DEL-on-upd-T 500B/e => YES, 1h, DEL-on-upd-T(tid) TD.load_tournament_director(tid,uid) 1d DEL-on-upd-TD 50B/e => YES, 1h, DEL-on-upd-TD(tid) TLP::load_tournament_ladder_props(tid) 1d DEL-on-upd-TLP 100B/tid => YES, 1d, DEL-on-upd-TLP(tid) TPR.load_tournament_properties(tid) 1d DEL-on-upd-TPR 50B/tid => YES, 1d, DEL-on-upd-TPR(tid) TL::load_tournament_ladder(tid) 1h DEL-on-upd-TL 40B/tid*entry => YES, 1h, DEL-on-upd-TL(tid) { join, rm-user, chall, g-end, etc }; SOME C-data TG::load_tournament_games(tid,Round) 1h DEL-on-upd-TG 70B/tid*entry => YES, 1h, DEL-on-upd-TG(tid,*); SOME C-data TRND::load_tournament_round(tid,Round) 1d DEL-on-upd-TRND 80B/tid => YES, 1d, DEL-on-upd-TRND(tid,*) TPOOL::load_tournament_pools(tid,Round) 1h DEL-on-upd-* 20B/tid*entry => YES, 1h, DEL-on-upd-TP/P(tid,*) !? TD.load_tournament_directors(tid) 1d DEL-on-upd-TD/P 50B/e => YES, 1h, DEL-on-upd-TD/P(tid) TP.count_tournament_participants(tid) 1h DEL-on-upd-TP 50B/tid => YES, 1h, DEL-on-upd-TP TN.load_tournament_news(tid) 1d DEL-on-upd-TN 1KB/tid*entry => YES, 1d, DEL-on-upd-TN TRES::load_tournament_results(tid) 1h DEL-on-upd-TRES/P 50B/tid*entry => YES, 1h, DEL-on-upd-TRES # NO prio (for now) User.load_quick_userinfo(uid) 1h DEL-on-upd 50B/e faq.find_entries(id) 1d DEL-on-upd-ed/tr 1KB * 200 userinfo.find_bio(uid) 1d DEL-on-upd-user 200B/user userinfo.count_gameplayer(uid) 1d DEL-on-user-upd 10B/user waiting_room.find_waiters(uid) ? DEL-on-upd-wr/gend/cr-g/c-sysfl 3KB/user -- cache std-search only Forum::load_forum_list(uid) 1h DEL-on-forum-post ?KB/user -- load forum-list with FR-user Forum.load_threads(forum_id,uid) -- load forum-threads for fid + FR-user ForumThread.load_posts([forum_id],thread_id) -- load thread-posts GamePlayer::load_uid_for_move(uid,col,ord) 1d DEL-on-upd 10B/e NextGameOrder::load_game_priority(gid,uid) 1d DEL-on-upd-user 10B/e Shape.load_shapes(-) 12h DEL-on-upd 4KB people.find_faq_admins(-) 1d DEL-on-upd-adm 100B people.find_forum_moderators(-) 1d DEL-on-upd-adm 100B people.find_translators(-) 1d DEL-on-upd-adm 1KB Profile::load_profiles2(uid,Types) 1h DEL-on-upd-user 200B/user -- not for templates TN.load_tournament_news(tid) 1d DEL-on-upd-TN 1KB/tid*entry # TO-CHECK rss/status.php quick_status.php # CHECK again (why is this not appearing in logs) ? CHK statisticspng.min_max(-) 1d DEL-on-upd-cron 50B => YES, 1d, EXPIRE CHK statisticspng.load_data(-) 1d DEL-on-upd-cron 200KB => YES, 1d, EXPIRE CHK statratingspng.query0(-) 1h DEL-on-upd-cron 200B CHK statratingspng.query1(-) 1h DEL-on-upd-cron 200B CHK statratingspng.query2(-) 1h DEL-on-upd-cron 200B #-------------------------------------------------------------------------- # check all below PHP-files (application) # C=strong candidate, '-' = no candidate, E = edit-page (no candidate) # check E features/edit_feature.php E features/vote_feature.php features/list_features.php features/list_votes.php E forum/admin.php forum/admin_show_forumlog.php forum/index.php forum/list.php forum/read.php forum/search.php - goodies/index.php - js/index.php - pattern/index.php - rss/index.php rss/status.php - scripts/experiments/activity_players.php - scripts/experiments/analyze-slow_log.php - scripts/experiments/bitfield_mysql_set.php - scripts/experiments/bitmask_signed_int.php - scripts/experiments/grabout.php - scripts/updates/fix_game_comments-1_0_15.php - scripts/updates/fix_message_thread-1_0_15.php - scripts/admin-export-games_info.php - scripts/admin-sgf-bulk.php - scripts/apc_cache_info.php - scripts/apc_clear_cache.php - scripts/browser_stats.php - scripts/check_block_ip.php - scripts/check_players_email.php - scripts/clear_datastore.php - scripts/convert_posindex.php - scripts/data_export.php - scripts/fix_game_snapshot.php - scripts/fix_games_timeleft.php - scripts/forum_consistency.php - scripts/game_consistency.php scripts/generate_translation_texts.php - scripts/index.php - scripts/integration-tests/seed_db.php - scripts/mailtest.php - scripts/make_all_translationfiles.php - scripts/make_handicap_pattern.php - scripts/message_consistency.php - scripts/phpinfo.php - scripts/player_consistency.php - scripts/recalculate_ratings2.php - scripts/server-info.php - scripts/sql_game_export.php - scripts/start_frozen_clocks.php - scripts/tournament_consistency.php - scripts/translation_consistency.php - scripts/update_translation_pages.php - scripts/voting_consistency.php - skins/dragon/index.php - skins/dragon2/index.php - skins/index.php - skins/known_skins.php tournaments/ladder/admin.php tournaments/ladder/challenge.php tournaments/ladder/edit_props.php - tournaments/ladder/index.php tournaments/ladder/withdraw.php tournaments/ladder/view.php tournaments/roundrobin/create_pools.php tournaments/roundrobin/define_pools.php tournaments/roundrobin/edit_pairing.php tournaments/roundrobin/edit_pools.php tournaments/roundrobin/edit_ranks.php tournaments/roundrobin/edit_round_props.php tournaments/roundrobin/edit_round_status.php tournaments/roundrobin/edit_rounds.php - tournaments/roundrobin/index.php tournaments/roundrobin/view_pools.php tournaments/cron_tournaments.php tournaments/edit_director.php tournaments/edit_lock.php tournaments/edit_news.php tournaments/edit_participant.php tournaments/edit_properties.php tournaments/edit_rules.php tournaments/edit_status.php tournaments/edit_tournament.php tournaments/game_admin.php - tournaments/index.php tournaments/list_directors.php tournaments/list_news.php tournaments/list_participants.php tournaments/list_tournaments.php tournaments/manage_tournament.php tournaments/register.php - tournaments/show_tournament_log.php tournaments/view_news.php tournaments/view_tournament.php - tournaments/wizard.php - wap/index.php wap/status.php admin.php - admin_admins.php admin_bulletin.php admin_do_translators.php admin_faq.php admin_game.php admin_password.php admin_rating.php admin_show_adminlog.php admin_show_errorlog.php admin_show_faqlog.php admin_show_users.php admin_survey.php admin_translators.php admin_users.php - bookmark.php change_bio.php change_password.php clock_tick.php confirm.php - cron_chained.php daily_cron.php do_registration_blocked.php - docs.php - donation.php - donation_success.php edit_bio.php edit_bulletin.php edit_contact.php edit_folders.php edit_password.php edit_picture.php edit_profile.php edit_rating.php edit_shape.php edit_vacation.php - error.php faq.php forgot.php game.php game_calc.php game_comments.php game_editor.php game_players.php game_rematch.php gameinfo.php goban_editor.php halfhourly_cron.php index.php install.php introduction.php join_waitingroom_game.php licence.php links.php list_bulletins.php list_contacts.php list_messages.php list_shapes.php list_surveys.php login.php message.php message_thread.php new_game.php news.php opponents.php people.php - policy.php - quick_do.php - quick_play.php quick_status.php rank_converter.php ratinggraph.php ratingpng.php register.php search_messages.php send_new_password.php sgf.php show_games.php site_map.php snapshot.php statistics.php statisticspng.php statratingspng.php status.php templates.php - todo.php translate.php userinfo.php users.php view_bulletin.php view_shape.php view_survey.php waiting_room.php #------------------------------------------------------------------------ # check all below PHP-includes features/lib_votes.php forum/class_forum_options.php forum/class_forum_read.php forum/forum_functions.php forum/post.php include/db/bulletin.php include/db/games.php include/db/index.php include/db/ratingchangeadmin.php include/db/ratinglog.php include/db/shape.php include/db/survey.php include/db/survey_option.php include/db/survey_vote.php include/db/waitingroom.php include/quick/quick_bulletin.php include/quick/quick_contact.php include/quick/quick_folder.php include/quick/quick_game.php include/quick/quick_game_info.php include/quick/quick_game_list.php include/quick/quick_handler.php include/quick/quick_message.php include/quick/quick_suite.php include/quick/quick_user.php include/quick/quick_wroom.php include/admin_faq_functions.php include/board.php include/cache_clock.php include/classlib_bitset.php include/classlib_goban.php include/classlib_matrix.php include/classlib_profile.php include/classlib_upload.php include/classlib_user.php include/classlib_userconfig.php include/classlib_userpicture.php include/classlib_userquota.php include/config-local.php include/config.php include/connect2mysql.php include/contacts.php include/coords.php include/countries.php include/db_classes.php include/error_codes.php include/error_functions.php include/faq_functions.php include/filter.php include/filter_functions.php include/filter_parser.php include/filterlib_country.php include/filterlib_gametype.php include/filterlib_mysqlmatch.php include/form_functions.php include/game_functions.php include/game_texts.php include/gamelist_control.php include/globals.php include/goban_handler_dgsgame.php include/goban_handler_gfx.php include/goban_handler_sl.php include/graph.php include/gui_bulletin.php include/gui_functions.php include/gui_user_functions.php include/index.php include/json_pear.php include/languages.php include/mail_functions.php include/make_game.php include/make_translationfiles.php include/message_functions.php include/move.php include/page_functions.php include/quick_common.php include/rank_converter.php include/rating.php include/register_functions.php include/sgf_builder.php include/sgf_parser.php include/shape_control.php include/std_classes.php include/std_functions.php include/survey_control.php include/table_columns.php include/table_infos.php include/time_functions.php include/timezones.php include/tokenizer.php include/translation_functions.php include/utilities.php include/wroom_control.php tournaments/include/index.php tournaments/include/tournament.php tournaments/include/tournament_cache.php tournaments/include/tournament_director.php tournaments/include/tournament_extension.php tournaments/include/tournament_factory.php tournaments/include/tournament_games.php tournaments/include/tournament_globals.php tournaments/include/tournament_gui_helper.php tournaments/include/tournament_helper.php tournaments/include/tournament_ladder.php tournaments/include/tournament_ladder_props.php tournaments/include/tournament_limits.php tournaments/include/tournament_log.php tournaments/include/tournament_log_helper.php tournaments/include/tournament_news.php tournaments/include/tournament_participant.php tournaments/include/tournament_pool.php tournaments/include/tournament_pool_classes.php tournaments/include/tournament_properties.php tournaments/include/tournament_result.php tournaments/include/tournament_round.php tournaments/include/tournament_round_status.php tournaments/include/tournament_rules.php tournaments/include/tournament_status.php tournaments/include/tournament_template.php tournaments/include/tournament_template_ladder.php tournaments/include/tournament_template_round_robin.php tournaments/include/tournament_utils.php tournaments/include/types/dgs_ladder.php tournaments/include/types/dgs_round_robin.php tournaments/include/types/index.php tournaments/include/types/private_ladder.php tournaments/include/types/public_ladder.php #-------- (B) Future Enhancements --------------------------------------------- * NTH: cache-bypassing with URL-argument: A nice-to-have feature for the caching, which is not implemented yet would be to allow certain admins to be able to bypass the caching by providing a specific URL-parameter like 'bypass_cache=1' for example. This could be controlable with the field Players.AdminOption.