Project

General

Profile

Actions

Feature proposal #7454

closed

Implement stored caching for streams

Added by krileon almost 6 years ago. Updated over 3 years ago.

Status:
Rejected
Priority:
High
Assignee:
Target version:
-
Start date:
04 February 2019
Due date:
% Done:

100%

Estimated time:

Description

Streams can't be retrieved by their ID except by the person who generated it since it's tied to their session (so each user has their own copy). Implement a database stored usage of this so it is no longer tied to session (causing session bloat) and it can be linked since it will have a static ID. Ideally it'd be something like the following.

id | user_id | type | asset | params

The streams parameters would be stored in params and with type can handle all the stream types. This could result in A LOT of rows though so maybe some sort of wildcard would be appropriate or possibly doesn't need to know user_id at all. Needs testing and further research, but this would allow for share URLs and much more.


Related issues 1 (0 open1 closed)

Related to CB Activity - Feature proposal #8009: Rewrite some assets with appropriate pointersRejectedkrileon12 May 2020

Actions
Actions #1

Updated by krileon almost 6 years ago

Best way to also handle this is rewrite how stream objects are used. Basically transition to them being read only and no longer set new values in the stream. Once the stream is created it's cached and isn't altered from then on. Need to also have a means of reusing the caches as having 1 cache per user profile stream would be excessive so additional parameters will likely need to be added to URLs and passed around (e.g. user, paging parameters, and any filters).

The entire idea behind this is to allow a single id to rebuild the streams parameters so we aren't trying to pass 20 something parameters between pages. This also allows rebuild of the stream from its id.

Actions #2

Updated by krileon almost 6 years ago

  • Status changed from Assigned to Rejected
  • Target version deleted (6.0.0)
  • % Done changed from 0 to 100

Passing the parameters between sessions is the most reliable and safe way to do this. It completely prevents tampering as well. Using a separate table results in massive read/write access hurting performance for little benefit. The share functionality should just be able to safely reverse an activity/comment id (as it already somewhat does).

Actions #3

Updated by krileon over 5 years ago

  • Status changed from Rejected to Assigned
  • Target version set to 6.0.0
  • % Done changed from 100 to 0

The best solution for this is to stop generating streams all together. Create a new database object and a new database table. Streams are then created beforehand, but are generic and reusable. So for example all profile streams would just be 1 stream. This means a simple ->load( StreamID ) would load the stream and prepare its parameters. Asset parsing would then be done to handle the substitutions behavior. This would need to have a solution for things like CB Gallery, etc.. streams where we need data from CB Gallery to set the asset, but that can be resolved with some object passing behavior.

In short this eliminates session access and bloat. It eliminates complicated changing a streams configuration. It then also makes Streams just another table object.

You'd then just select the stream you want to load when outputting a field, menu entry, etc.. Needs a lot of though as the asset parsing for objects like CB Gallery media object will be tricky.

Actions #4

Updated by krileon over 5 years ago

Basically instead of having stream parameters all over the place you'll have 2 locations. The globals in CB Activity > Parameters and the Streams within CB Activity > Streams. When creating a Activity field all you'd supply is the Asset and select what Stream to use. The only data that'd need to be passed around through the session is the Asset at that point instead of the entire streams worth of parameters.

This would have serious B/C implications, but needs to be considered to improve the usability of Streams as more features are added and the performance.

This could be taken slightly further with a stream column added to database tables so entries keep track of what stream they came from. This way when an entry is loaded inline or attempted to be accessed by its entry id it can reload its stream parameters perfectly fine.

Actions #5

Updated by krileon over 4 years ago

Actions #6

Updated by krileon over 4 years ago

With note of the asset changes in #8009 stream assets should no longer be allowed to be overridden. This means the asset needs to be defined at time of stream creation. So for example you'd have a stream with the following asset.

profile.[target_id]

This also means stream overrides go away as well (except those that are purely for display).

The idea here is we should only need to pass either the activity id, comment id, etc.. or the stream id with target id and object id. So for existing posts all endpoints would just accept the activity id and since the stream the activity is associated with is stored with the activity we can rebuild it from just that activity id. For posts that don't exist yet we pass stream id, target id, and object id which can then just be substituted in. Standard substitutions will continue to be supported as well.

When it comes to what part is the object and what is the target the following will be used as proposed in #8009.

TARGET.TARGET_ID.OBJECT.OBJECT_ID

If only 1 pair is provided. For example the following.

profile.USER_ID

So in storage we'd have for example the following. We don't have an object in this case as it's a regular post.

verb: post
target: user
target_id: USER_ID

Another example is a GJ post with the below.

groupjive.group.GROUP_ID

This would just be as follows.

verb: post
target: group
target_id: GROUP_ID

Then you've a CB Gallery entry on GJ as follows.

groupjive.group.10.gallery.photos.30355

That would be something like the following.

verb: upload
object: photo
object_id: PHOTO_ID
target: group
target_id: GROUP_ID

Primary issue is how to handle comment streams of activity posts on an activity stream. Most likely the best way to deal with this you no longer define the parameters of those sub-streams in the activity stream, but instead select a comment stream from a dropdown. This doesn't really answer the question of how to handle those sub-stream assets. For example comments on activity posts made on profiles are just tied to that post, but comments on activity post made on gallery media are tied to that media.

Actions #7

Updated by krileon over 3 years ago

For comments on activity entries it should use the following structure.

verb: comment
target: activity
target_id: ACTIVITY_ID

Object should be in respect to whatever the activity object is. So for example if it's a comment on a post in a group you'd have the following.

object: group
object_id: GROUP_ID

If it's a comment on a activity entry on a profile stream you'd have the following.

object: user
object_id: USER_ID

The main issue comes from activity entries about a CB Gallery photo since we want the modal window AND the activity entry to share comments. So would likely need to adjust what comment stream is used on a per-activity level.

Actions #8

Updated by krileon over 3 years ago

Next issue is determining what activity is shown. Currently we're using SQL wildcards. So if you use an asset of profile it adds asset = "profile.42" OR user_id = 42 to the query. Need a way to directly configure exactly what should show. Maybe use a repeat usage where you supply target, target_id, object, and/or object_id to add to the query in addition to adding user ids. So for example the following.

target | target_id | object | object_id
user | [user_id]

The above would be the same as asset = profile.42 and filter by a target = user and target_id = [user_id]. A repeat usage for Include Activity and Exclude Activity with this behavior could be used. Next you'd have an Include Users and Exclude Users parameter to supply user ids. Example as follows.

Include Users: [user_id]

This would show all that was created by the user as well as all activity targeting the user, but how do we handle showing connections and following activity here which are special asset cases? I suppose Include/Exclude Connections and Include/Exclude Following parameters could work.

Note these parameters help define ADDITIONAL activity to show/hide. When configuring the stream you must configure target, target_id, object, and/or object_id which will be the behavior used for storage.

Actions #9

Updated by krileon over 3 years ago

Should a type column also be considered? Example as follows.

type | verb | target | target_id | object | object_id

post | create | group | GROUP_ID
photo | create | user | USER_ID | photo | PHOTO_ID

Not entirely sure it's necessary, but worth considering. The below probably still makes the most sense.

verb | target | target_id | object | object_id

post | group | GROUP_ID
join | group | GROUP_ID
leave | group | GROUP_ID
create | group | GROUP_ID
upload | user | USER_ID | photo | PHOTO_ID

Actions #10

Updated by krileon over 3 years ago

Get rid of global stream parameters as well. You will no longer control stream settings from within CB Activity > Parameters. They will be entirely controlled within CB Activity > Streams. This means there is ONE location for controlling stream parameters and eliminates confusion.

Actions #11

Updated by krileon over 3 years ago

Need to figure out the best way to optimize displaying activity from multiple streams.

First you'll configure the target, target_id, object, and object_id with each being optional (e.g. target = user and target_id = [user_id]). This would result in both of those filters being added to the database query AND this will always be used for storage. Next have a tab with the following parameters.

Include All Activity: Yes/No

If the above is set to Yes then HIDE all of the following INCLUDE parameters for ALL usage. It will still be possible to exclude. This covers our Recent Activity page requirements.

Include Streams: include all activity with the selected stream ids (should this be comma list so we can use substitutions?)
Exclude Stream: exclude all activity with the selected stream ids

Include Users: include all activity created by the list of user ids
Exclude Users: exclude all activity created by the list of user ids

Include Users Connections: include all activity created by the list of user ids connections
Exclude Users Connections: exclude all activity created by the list of user ids connections

Include Users Following: include all activity the list of user ids is following
Exclude Users Following: exclude all activity the list of user ids is following

Include Verbs: include all activity created with the supplied list of verbs
Exclude Verbs: exclude all activity created with the supplied list of verbs

Include Activity: include all activity matching the supplied target, target_id, object, and object_id
Exclude Activity: exclude all activity matching the supplied target, target_id, object, and object_id

Maybe these should just be Filters with the first filter representing All? This would allow the entire tab to be named Filters and be used to represent what's displayed outside of the streams initial target, target_id, object, and object_id. All of these parameters need to support substitutions. The additional filters (not All) should be a cbrepeat usage so you can add as many filters as you like.

As for how to pass data to a stream we'll just goahead and include it with the stream output (e.g. data-stream="{id:ID_HERE,target:\"user\",target_id:\"42\"}"). This will be cleaned, loaded in, and access checked before anything happens anyway so it's perfectly fine to just do it this way. So for example CB GroupJive would use the following usage.

target: group
target_id: [target_id]

This would tell us the stream expects a target id to be set. If one isn't set fail the stream loading. When the stream is rendered CB GroupJive would give the group id to the stream to be used for [target_id]. When dealing with form POSTS, paging, etc.. we'll just pass the same target_id along with that data. When the stream loads GJ will verify the user has access to the group so tampering with this data won't cause any issues. Another option is to, and this is likely preferred method, use 2-way encryption using site key. Encrypt the JSON (using openssl_encrypt or something similar), set it in URLs as the stream id (e.g. &stream=ENCYRPTED_JSON), then decrypt in CB Activity. That'd be the safest and easiest option. Note if a target, object, etc.. is NOT set AFTER loading a stream then don't bother with the encryption and just use the actual stream id. This will keep most common streams faster since they won't need to bother with encryption, but again we probably don't even need to bother with doing this.

As for sharing activity via URL all we need is the id of the activity entry. The stream the activity belongs to will be stored. So we'll rebuild from that easily. Originally it was planned to support generating public share URLs, but with this method we don't need to.

The above filtering should probably still use Asset, but in code we explode it to target, target_id, etc.. otherwise the user could be responsible for a lot of parameters and could become very confusing.

Actions #12

Updated by krileon over 3 years ago

  • Status changed from Assigned to Rejected
  • Target version deleted (6.0.0)
  • % Done changed from 0 to 100

See #8663

Actions

Also available in: Atom PDF