User:Most2dot0/Ideas

Database

Many of the ideas scetched out below have already some implementation. I will link to the Modules, Data, etc. that realize some elements.

Common Database Structure

Have a common database structure for video sources (and possible other content, like pictures, e.g. FB & Insta posts w/o video, etc.). The data would be identified by URL, and include

URL
Source type (YT, IG, FB, ...)
Channel name & id (as subcategory of source type)
Language
Comment
Recording and publish dates
Open Graph data derived from the URL, especially all basic tags, i.e. Title, Description, Image

This would be stored in it's own table. Possibly the Open Graph data would even have it's own table, so that it is easier to regenerate.

The songs table would not repeat this information, other the ones needed for linkage. Instead, a table of events would be added with the song title as link to the songs database. Events could be specific rehearsal sessions, concerts, album (recordings), etc. The Events should be uniquely identifiable by name. Another table the would list recordings linked with the song title to the songs table, the event name to the event table, and with the URL to the video source info table.

Performances in the sense of a single song performance, in general as part of an event.

Implementation of Elements

The following data may bestored in their own wiki articles per entry. They can be identified to be ours such data by being part of an according category (though consider if a location would be directed to a section in an event article):

Songs
Events
Locations
Artist

It could be for the future considered to have templates there to tag available data for scraping purposes, so that e.g. a bot could periodically collect them into tables for use in other pages as well.

The following data may be stored in data formats like JSON, with a single page with a list of entries:

SongsToVideos
(Video)MetaData
Performances (with links to song, location, video-segements)

That the SongsToVideos table contains links to video-segments is an interim solution, which should not be needed in the long run. Instead, I envision the Performances table to be the central data storage that links all the other elements together. It will be augmented by the autogenerated MetaData table providing additional info for linked videos.

Performances Table

An entry in the performances table does not necessarily have its own name. It will link to other database elements by name, and can by identified by the combination of those, plus it's data. These are:

Song Title
Event
Date (possibly only as month or year indication)
Type (Music-Video, Track, Recording, Live, Soundcheck, Rehearsal, Sing-along, Lip-sync)
Position (e.g. on setlist of concert, or sequence number of different rehearsals of same title/event)
Venue (if known)
Comment
Video-Segments, with

- URL,

- start & end times (possibly only contained in URL)

- Duration (full, short, fragment),

- Quality (best, good, acceptable, poor)

Identification of Songs

Problem

Current Solution

Use the titles according to 1., which will make their use most easy and consistent with the rest of the wiki. The question is then, if it is required to offer the core title, and it's disambiguation seperately for display purposes?

If we decide for reasons of simpliciy that it is not required, we could have these three fields:

Title (soley used for technical identification)
Disambiguation (used for display)
WPext (just the disambiguation to form WP links)

For display purposes, we could then print

Title (Disambiguation),

whereas Wikipedia links would be formed as

{{w|Title (WPext)}}

If no WPext is given, the Disambiguation would be used in its place.

The mistake that is made is, that e.g.

All of Me (jazz standard)

will be misprinted as

All of Me (jazz standard)

On the other Hand, this has the added benefit of making it clear what is needed to identifiy it in this Wiki.

Note: the new Cargo based implementation of the {{SongsTable}} used in Songs does output "All of Me (jazz standard)".

Quality Indicators

Present some sort of "quality" indication, possibly regarding (either sepearate or maybe integrated) in

performance type (studio, live, rehearsal, sing-along, lip-sync),
recording length (full, short, fragment), and
audio quality (best, good, aceptable, poor)

These could possibly be integrated into a common 10-1 scale, calculated e.g. like this:

Base by Type: "studio"=8, "live"=8, rehearsal="6", "sing-along"=2, "lip-sync"=0
Length: "full" +/-0, "short" -1, "fragment" -4
Modify by Quality: "best" + 2, "good" +/0, aceptable "-2", poor "-4"

Limit result to 1-10.

Use of Database

The above information can be linked to e.g. display table listings for Channels, Events, Songs, etc.

Ideally there should be a common generic list generator, that can be configured for data (columns) displayed, and with arbitrary filters for each data type (column contents).

There could be templates to generate references (e.g. for footnotes) simply by providing the URL. (note {{cite by url}}, which will not scale)

Other DB usage ideas

Dates

Besides the idea of having specific event entries (see {{de}}, discussed here), we could simply link the existing {{da}}, {{d}}, and {{du}} to a table to collect dates mentioned. We could e.g. display the linked dates that have no anchor yet. or use those in the Cargo based calendar, to have less effort compared to using {{de}} in the timeline pages.

Tasks, notes, Kanban board

A template with (optionally) no visual ouput could be put onto pages that describe ideas or tasks. It could have an anchor created, so it could be directly jumped at.

This could also cover existing macros that denote article problems by adding them to categories, like "citation needed".

This way overviews of those spread notes are easily creatable. Entries could support a working status, e.g. similar as used on a Kanban board^↗, and displayed as one, so people can grab a task and indicate that they are actively working on it.

Cargo Implementation

It's being looked into moving the data from the big JSON files into the song's articles via the Cargo Extension. On reason is, that Wikimedia is inefficient in regards to storage when handling changes to big files. But Cargo also offers addtional features, like the possibility to browse the data without predefiend queries and with dynamic filters.

Database tables

The basic structure described above can be retained.

Database storage

The entries (rows) of several tables can be defined in the article page of the related song. It will contain several Cargo templates for the Song, the song Performances, and the Videos for each Performance. The song's Cargo template might be integrated into an existing song templates, like the {{Infobox song}}. The songs listing in the Songs article could then be generated using a Cargo query.

As alternative, the songs' Cargo templates might be integrated into the Songs article instead.

Events, Contexts, and Categories

As discussed in Data_talk:Performances.json#Splitting_event_field, there is the plan to have a hierachial context description instead of event entries in the database. This has been already implmented in some form in the current User:Most2dot0/Performances-devel prototype. The feedback on this prototype pointed into the direction, that verbatim links to events should rather point to wiki articles about them, as opposed to the event summary in the database. Links to such database summaries could still be realized by using icons instead. In the meantime, the [User:Most2dot0/Performances-devel]] prototype was updated to link to both articles and summaries, using typically an icon to indicate the later.

The prototype also showed, that there is a conflict of a structured hierachial context description with well sounding Event names. E.g, "Summer concert tour, 2022" vs "2022 Summer concert tour". With the use of Cargo this could be solved by adding Cargo entries for event names, linking them to the hierarchy description. E.g., in such an article about an event a Cargo template could be called like {{Context|Concert#Summer concert tour#2022}} to link that page to the context hierarchy used in the Performance database and the context entries used in the song's articles.

This a similar to the category wiki concept. So it might be possible to represent the context hierarchies also with nested wiki category entries. The above mentioned {{Context|...}} could also set the category of the lowest level include, which here would be "2022". But this reveals a conflict: while within the context idea, the could be several "2022" subcontexts in different branches of the hierarchy, in the wiki category system it would be the same "2022" category, linked to several higher categories. This could be resolved by using the full context hierary in the name of such a category, so e.g. "Concert#Summer concert tour#2022" instead of just "2022". This does not look "natural" in terms of wiki categories, though. Possibly, this could also be seen as a chance instead of an issue: a "sub-context" like the "with guitar" class would be interpreted as the specific branch in Cargo's context terms, but would cover e.g. both "Covers" and "Snippets" in terms of the wiki category idea. Though that might be something, we would rather solve with a "tag" system, anyhow.

Transition

Bulk edits

The distributed storage to be achieved with Cargo will likely be more difficult to maintain in regards to bulk changes to data. Unlike with the JSON pages, it's not possible to do bulk modifications with simple regex search and replace calls. Instead, it would likely require scripting or use of robots. For this reason, the database structure and most of the existing data should be in the desired final form before making the transition.

Possiple "preload" feature use

To easily build the Cargo data, the preload feature could be utilized for a preloaded section addition, by either the normal or a special version of the JSON based Performances listings, which could preload a template that then optains the required data from the JSON tables, and replaces itself with Cargo template calls for the data.

In the meantime, the JSON data has been auto-translated into a single page with cargo template calls, which is used already used in the current Performances prototype. In this form, bulk edits can still be done easily.

As an alternative to the preload approach, it would also be possible to manually move the data pieces to the songs article.

Code base reuse

Once the Carga database is complete, the JSON data could be reconstructed from it, and it should be possible to keep most of the existing presentation code if still desired or for a transitional phase.

The first implementation has shown that this is very feasible for longer use. Also, the drill-down interface is less usable than expected, and the formatting options to native Cargo queries are limited, so they likely are no alternative to Lua based table generation.

Saving storage space immediately

To gain immediate effects of easier editing of performance entries, plus reducing the storage requirements for changes, we could split this page, e.g. by the starting character of the song title, to have smaller individual pieces which create less overhead when edited.

Current Status

2025-10-25

All missing song articles were created, the ones for the cover songs with short introduction text, the ones for the originals just with preload-templates, unedited. The performances overview table listings have been changed to use the new production version of the Cargo database.

Unfortunately, apparently the "with"/"partner" fields were lost in the process, likely because of the difference in name. I likely need to see how to transfer them ...Devel tables most effectively.

2025-10-20

Due to the feedback the decision is made to go with the option to continue the rollout to individual song articles. Here is a summary of the feedback points that need special attention and futher preparation doing this:

Preload songs info: disambiguation, info and comment info from songs.json are to be pre-loaded as html comments, while for new song articles the disambiguation can be added to the infobox arguments.

Help and examples: some info is need to help how the new templates and support tools are to be handled.

2025-10-19

The development version Module:Performances-devel has been merged back into the production version Module:Performances, so that now supports Cargo tables. The production version of the Performances listing, Performances, has been modified to use the development tables of the Cargo database. The idea was that it can be updated with less overheadd by the addition of a 2nd .cargo page Performances-devel2.cargo for additions; however, for doing modificaions of existing entries, inlcuding adding alternative videos for already listed performances, this is no help. Thus current transient state has little benefit compared to the JSON based solution, but only makes changes more difficult, and it should be moved in any of these possible directions soon:

Quicky adding the performance and video Cargo template calls to the song articles, creating all missing ones, fully creating the upcoming production version of the Cargo databse. That was the original plan, and still is my prefered one, as it likely ensures best that the song articles and the entries in the database are in sync.
Going back to the single Cargo file Performances-devel.cargo. There could be links in the Performances to directly jump to the section containing a song's performances, and editing the only a subscection makes thinks already easier compared to the JSON based solution. It also has the benefit that bulk editings (like a change of the context structure) could still be easily done by working on a single file. It has the downside that it won't be saving storage space.
Instead of a single Cargo page, split it by starting letter in smaller chunks. This would make small edits take a magnitude less storage, while the direct jumps to the appropriate song section for editing could still be generated. It would make bulk editting more complex, though, so possibly instead of creating like 26 entries for A-Z we should go with like 10, grouping some not so pupulated starting letters. That way, bulk edits could be done by working externally on a joined version, and then be manually split and copied back into those chunks again.

2025-10-09

Performances and Videos Cargo tables habe been created via the Template:Performance/Template:Video Cargo templates. The use Template:xl to form links for display. In addition, a modified version of Template:ytv, Template:cytv (Cargo YouTube Video) has been created, which calls the Video template in addition to doing the ytv stuff. To work, the performance reference id ("pid") of the related performance has to be added as a named parameter.

Note, that the display can be changed. E.g. currently the performance template displays the event field, this will be changed to a proper display of the context (occasion, occurance, segemnt) in the future. Also there is a bug, that comments do't get stored with the cytv template, which could be fixed later.

A modified version of the "All songs by title" table has been created, Preload-links with performances listings per song". It contains added links to either append Cargo templates for the performances (with videos) of the song ("(+)", for existing articles), or to create new articles ("C") by the use of templates. Upon first save of the preloaded page data, the templates required to recreated the data in the production version databae are being generated and saved. Currently there is only a template to create cover song articles, but that will change, and it is expected that choosing the right template for article creation can be automated.

So far, all existing articles for song starting with the letter "7", "A", and "B" have been updated with Cargo data. In addtion, all missing cover song articles for songs starting with "A" have been created.

2025-10-04

The User:most2dot0/Performances-Devel prototype has been sucessfully modified to utilize data stored in the Template:Performance-devel and Template:Video-devel Cargo tables. For this purpose, the Module:Performances-devel was modified to be feed with data from Cargo queries, which allow for joining data from different tables, in this case the videos info linked to the performances entries. Currently all the data for those is defined in the Performances-devel.cargo article. It is expected, that in the production version these will be distributed to the song pages covered by those performances/videos.

Embedded YouTube Player

The player already features easy configuration, arbitrary start/end time for videos in a playlist. The follwing improvements are being considered

Further Improvements

Also Play FB, IG, TT Videos

This has been considered in the past, but is postponed, as the other social media don't offer a comparable API as YouTube does, so many features would lack.

Repeat Options

Enable no autoplay, repeat single clip, complete playlist once, complete playlist loop (this is current fixed mode).

Consider if at the end of a playlist, the player could start the next one on the page (might be possible through the numbered IDs each player gets assigend when being substituted)

Player Indication at URL

Consider to indicate e.g. via color coding / added icon /changed icon that an URL will start the player instead of following the link to the source.

Floating player that survives inter page jumps

Currently, the players are tied to a certain place on a certain page. And while they coordinate, so that every other play stops if a new one starts playing, they are seperate entities. When a link jumps to a different page on the wiki, the old page is stopped an with it the player. With a single floating player, that is fed by distributed playlist controls, and the use of Ajax, it would be possible to keep the player playing even when switching pages. This would be achieved by binding the embbedded player to the highest level of the html <body> element, and only simulating page changes by loading the target page contents and have them replace the old page's contents. This is not trival, but apparently could be achieved by a Gadget. A first attempt was made here, but e.g. back button action are not handled correctly.

Use to obtain video's meta data

Unlike assumed earlier, it's apparently not possible to request the video title and the channel name from the players API. So this could not be used directly to add them to a metadata database if they are still missing.

There would be alternatives: getting a YouTube API key for the wiki to use the "API v3", or to use the oembed^↗ endpoint YouTube provides. But apparently, this can not be queried client side, so that would require some setup in the wiki server as well. Both of these methods could be run perdiodically server side, or could be triggered client side e.g. by the YouTube player.

Hover previews for certain anchored link targets

Timeline entries

A first implementation of this was done with MediaWiki:Gadget-TimelinePreview.js Gadget, where the timeline entries of date links get shown in preview popups. It works by loading the target page in the background, for searching and copying the text to be previews at the jump target ("hash" value in URL, anchor on the page). This idea could be generalized for other usages.

Performance table entries

E.g. for jumps within or to the Performances tables, previews of the sub-tables could be shown, which for context links would list the songs sung there, or for song links show on which other events these were performed.

Note: that the underlying mechanism could also replace the on-demand Cargo queries of {{SongsTable}}, which likely puts less computational load on the server.

Anonymous

Search