Showing posts with label software. Show all posts
Showing posts with label software. Show all posts

Wednesday, February 05, 2025

Be Free to Comment – Heedfully

[7 Feb 2025: Added a bit more detail concerning the term 'heedfulness', with a link to my research.
21 Feb 2025: Added an abstract
24 Feb 2025: Added a word cloud, generated by Voyant Tools.]

Abstract

News media have for years allowed members of the public to add their comments on some of their published output (bulletins, reports, features, and so on), which we refer to as items. However, such opportunities to have your say have been gradually diminishing, raising concerns about bias and control of the narrative. Yet, with increasing divisions in society across the world, the need for freedom of expression, to learn other points of view and enter into dialogue so as to grow in understanding, is surely greater than ever.

To address the problem, we propose a new overarching service that will allow members of the public to comment on items published especially by professional news media from around the world. All comments brought together in one place, with links off to the original news sources. It will, in particular, where legally acceptable to do so, allow commenting on items (or summaries, where access to items is restricted) that do not yet have a comments area. It will also act as a knowledge base, pooling sources from the articles and the comments, allowing different voices, facilitating a neutral point of view (NPOV).

Such an undertaking requires a heightened level of heedfulness, attentiveness about the consequences of our verbal actions. We therefore propose guidelines for contributions, based on the Buddha’s teachings, on what we term ‘5-star speech’: truthful, offered in a mind of good will, expressed in a kind manner, something relevant and beneficial to the recipient, and at the right time.

An enhanced form of moderation is recommended to help maintain the guidelines. This is to be steered not by academics or those in the news media industry, but by those who are skilled in conflict resolution. Accordingly, we recommend as candidates for this role those who are trained in Nonviolent Communication or similar.

Introduction

I offer some initial thoughts for a novel service, a kind of overarching comment system for news media across the world.

First, we need some background.

Then we can introduce our intentions for the service.

A narrowing field

Ever since news organisations started publishing on the Web over 20 years ago, their current affairs content has steadily grown from front page headlines and news bulletins to in-depth articles and features, ranging from local to international affairs. Nowadays, with digital editions having largely overtaken print-based media, the majority of their output and subscribers are online.

Opportunities were soon identified for reader contributions through feedback mechanisms, discussion fora and online communities, following a practice that was established before the arrival of the Web with various Internet services such as bulletin board systems (BBS), Listservs and UseNet newsgroups. Eventually, news media organisations provided opportunities as part of their Web presence. Hence, the BBC’s Have Your Say and The Guardian’s Comment is Free, which allowed members of the public to share their views on a variety of topical issues.

The Guardian usefully provides various entry points to their conversations, including an index of themes. Although it was last updated in 2013, it still links to a list of current articles on the given theme, such as Christianity. However, relatively few articles have a comments count, whereas clicking on the same link 11 or 12 years ago (archived), reveals a page where almost every article has comment counts.

This reflects a general pattern. In recent years, the number of articles open for comment has dwindled as has the visibility of comments — in some cases, only succinct emails are solicited. The media companies have become more selective, increasingly erring on the side of caution, particularly on matters that they regard as sensitive (to whom, is not always clear).

And this change has been noticed:

(Google's auto-complete suggestions on entering a search about The Guardian's service draw on existing usage patterns.)

Feelings of Discomfort as signs of Opportunities

Why the change? The inclination towards closure became increasingly expressed as more people joined in the conversation. Whilst at the first glance this was democracy in action, it also — to some — led to an unacceptable decrease in overall quality. Such a sentiment was expressed in 2015 by a journalist in The Guardian, who describes feeling under attack, where odious comments become prominent and worthwhile contributions become buried. Yet, as the article hints, avoidance can mean missing out on something valuable. And the issue of quality is not a new phenomenon: I recall a contributor to a Usenet group confiding to me in around 1995 that he was disinclined to post further because the signal-to-noise ratio had become too low, but he still continued to make contributions where there was something of interest.

Having forbearance can be worthwhile. Especially with newcomers. It did not take me long to get into a bit of hot water, receiving a couple of angry responses following my first Usenet posting (on ‘Quantum Theory and Meditation’). In hindsight, I was naive and not following good practice: the choice of topic (too ambitious), the way I expressed myself (perhaps dismissive in tone) and the use of markdown (what?!). Altogether probably injudicious. Within two days the first irate message appeared in my Inbox, “No! No! No! … ” , quite a sobering experience. Yet, I never intended to offend and, undeterred, I persevered in trying to understand and to articulate my point. The thread continued, gaining some further interest, others extended the conversation and I was happy to see it develop freely. I became on friendly terms with several participants and along the way received glimpses of the life of an atomic physicist working in a laboratory. Although I had got bitten, I found the experience valuable. Viewed as opportunities to learn, it is worth persevering with such situations and seeking to improve one’s conduct. Potentially everyone can benefit.

The problem with quality on the Web today is in large part due to the user interface, where the design is often optimised for ease of use. Following Steve Krug’s influential work, Don’t Make Me Think, Web designers have made it a trivial process to enter one’s thoughts in a box and hit the [Send] — the interfaces are slick, as in oil slick, with words potentially gushing forth without due consideration.

Freedom of Expression and the Right to be Heard

Much is made, especially in the West, of the right to freedom of expression, as declared in Article 19 of the United Nations’ Universal Declaration of Human Rights (1948)

Everyone has the right to freedom of opinion and expression; this right includes freedom to hold opinions without interference and to seek, receive and impart information and ideas through any media and regardless of frontiers.

(English language version; many other languages available via the the translation project.)

However, the International Covenant on Civil and Political Rights added a limiting clause to Article 19 (1966):

3. The exercise of the rights provided for in paragraph 2 of this article carries with it special duties and responsibilities. It may therefore be subject to certain restrictions, but these shall only be such as are provided by law and are necessary:
(a) For respect of the rights or reputations of others;
(b) For the protection of national security or of public order (ordre public), or of public health or morals.

Who decides what is necessary? What is really in the national interest? These clauses are open to varying interpretation, but in general there is a spectrum of views from those who declare themselves to be ‘free speech absolutists’ to others who would like it closely controlled.

In trying to foster appropriate contributions to Comment is Free, The Guardian weighs in with a quote from a former editor and owner of the Manchester Guardian, C.P. Scott, taken from his centenary essay:

“Comment is free, but facts are sacred.”

The subclause is heavily leaned upon as a restraint, but in reality it may not be a straightforward matter to establish those facts and what may be promulgated as such can sometimes fade away (consider, for example, the origins of the Covid-19 virus). However, it is soon followed by another sentence that also merits quoting:

“The voice of opponents no less than that of friends has a right to be heard."

This becomes more urgent in a society that has an increasing tendency to see complex situations in simplistic terms as black and white, which often creates or reinforces division. In that contentious area of ‘freedom of speech’, it is evident is that many people find it very hard to listen to other points of view and to discuss them with an open mind.

Balance and Impartiality

Some media outlets have maintained comment sections across a broad spectrum of topics and taken a relatively liberal position in terms of allowing critical views. To enable this they still maintain oversight and will promptly delete comments regarded as falling foul of their community guidelines. They often requiring participants to have a paid subscription. One example is the Daily Telegraph, which provides conversation space for perhaps the majority of their articles, with journalists occasionally joining in for a kind of real-time Q&A.

Yet, even there, certain topics or people still seem out of bounds, which cannot be explained by the fulfilment of legal obligations alone. So the spotlight is turned on networks and relationships, particularly those of a financial nature — who are the sponsors and what do they have to gain? The Daily Telegraph Global Health Security is funded by the Bill & Melinda Gates Foundation. Can the paper really retain full editorial independence on matters relating to health and the pharmaceutical industry?

Cases of vaccine harms emerged as soon as they were issued: UK CV Family, which has long-known about problems, was granted core participant status in module 4 of the UK's Covid-19 Public Inquiry in 2023, but had been reporting problems the year before with mRNA-based vaccines from Pfizer and Moderna, as well as AstraZeneca's viral vector vaccine . However, they only get mentioned by the Telegraph in 2024, and the article itself confines itself to AstraZeneca, after they had left the market. Gradual acceptance of Covid-19 vaccine harms and their seriousness is belatedly emerging: ‘The truth about Covid vaccines may finally be coming to light’, but the article is brief and comments conflict markedly with its general direction. Similarly, an investigation by a former Government minister has again focused solely on AstraZeneca. Newspapers are expected to investigate, to reveal different sides of the story, but have fallen short with this virus. It brings to mind the old adage, “He who pays the piper calls the tune.”

Many comments have made points to this effect, but for media that are supposedly on the pulse, the response on matters like this has been glacial. It tends to shift faster the more such valid data, properly established, become common knowledge. Providing a global forum for the public to share their responses and insights can help with this process.

Acknowledging and working with diverse views

The Internet has doubtless assisted many people in freedom of expression. However, it is generally recognised that there is increasing polarisation of views, fostering growing anxiety and talk of the need for more constraints (usually of others). Differences are evident in the political views expressed in the editorials of The Daily Telegraph and The Guardian, but are even more pronounced among their readers, who appear to form distinct camps. Attitudes to other points of view are often derogatory and dismissive (“LOL”); some [im]posters seize the opportunity to engage in trolling, searching for places to score points, often using language designed to upset. There’s little benefit in the short term and probably negative effects in the long term.

This dynamic is nothing new, but one can rise above it.

Or he might say: 'Whereas some recluses and brahmins, while living on the food offered by the faithful, engage in wrangling argumentation, (saying to one another): "You don't understand this doctrine and discipline. I am the one who understands this doctrine and discipline." — "How can you understand this doctrine and discipline?" — "You're practising the wrong way. I'm practising the right way." — "I'm being consistent. You're inconsistent." — "What should have been said first you said last, what should have been said last you said first." — "What you took so long to think out has been confuted." — "Your doctrine has been refuted. You're defeated. Go, try to save your doctrine, or disentangle yourself now if you can" — the recluse Gotama abstains from such wrangling argumentation.'

Brahmajāla Sutta (the All-embracing Net of Views), Digha Nikaya (Long Discourses of the Buddha) number 1, translated by Bhikkhu Bodhi

This quote describes the behaviour of spiritual wanderers 2,500 years ago. Today, across all walks of life, there are still these same futile patterns of argumentation, which brings the weariness that we’ve already indicated. However, the Buddha spent much of his time giving teachings, being engaged in conversation, in which he makes a valid case, draws on solid evidence and develops sound lines of reasoning. So, in that spirit, we can try to foster healthy, constructive discussion.

In fact, a proportion of readers cross over and subscribe to other papers of different hues and contribute comments there. There is great potential in this scenario. I remember being told about a reader with an avid interest in current affairs who regarded himself as a liberal, naturally inclined to The Guardian. However, he regularly read the Daily Telegraph to seek out other points of view and developed an appreciation of its “cosy” perspective without a fundamental shift in his views. He sounds like the kind of person able to enter dialogue with a relaxed openness that I would like to foster online.

The potential of collective support is also worth noting. In 1999, I attended a remarkable session of the annual conference of the International Interfaith Centre, Facing The Past – Freeing the Future — in which there was a wary dialogue between a Turkish Cypriot and a Greek Cypriot, both honed in arguing for their respective opposing positions. The audience, for a long time without speaking, tried to support them, willing them on to seek cooperation rather than confrontation.

At the last minute, as recorded by David Craig for the IIC Newsletter 11, June 1999 , there was a breakthrough:

“There was a tangible tension in the room but when Deepak Naik asked what each was going to do when he got back to his community, common ground had been found: each would tell the young people in his island group that the other side wanted peace, and co existence and an end to division.”

Sources

News media often omit citing primary sources, seemingly content to receive information by word of mouth, in press briefings, though a fairly recent trend has been to embrace ‘fact checkers’ to ostensibly give definitive answers. As to why this is the case, there has (of course!) been some discussion with suggestions that the Internet’s disruptive nature is difficult for them to handle.

From my experience, whereas the BBC makes reference to primary sources in some of its articles (with clear disclaimers, as it sees fit), I have found the Daily Telegraph hardly does so. Articles seldom contain a dedicated reference section (which might be expected at the foot of the article) and linked text that sounds like it should point to a source (such as “according to a recently released report”) usually leads instead to another Telegraph article, effectively a kind of clickbait. That might boost the service’s analytics and aid monetisation in the short term, but when repeated, I am put off.

With trust ebbing away from mainstream media, access to primary sources really ought to be given priority.

Outline of a Proposal

Here’s my idea. I propose Be Free to Comment — Heedfully, a new service reflecting the primary intention of freedom of expression conducted in an appropriate manner. It’s a working title to capture what it’s about. By ‘heedful’ I have in mind the Pali Buddhist term appamāda, a notion I have researched in relation to the Fifth Precept, concerning the avoidance of intoxicants. The heedfulness is self-directed, keeping in mind the consequences of one's actions, and should be in reference to 5-star speech, which is described below.

The service acts as a meta discussion area in which — subject to legal compliance in respective jurisdictions — each article of a mainstream newspaper has an accompanying discussion area with a link to the original source (which is automatically archived, for example, by the Wayback Machine). Where access to articles is restricted, for example, by requiring a subscription, a paragraph summary of the item is created as a basis for a broader discussion of the topic as a whole.

This is just an outline to indicate my approach.

Which Sources?

Following a well-established process, considering how the largest social media have evolved, the service should start on a small scale and be expanded gradually.

So, it might begin in the UK, with a few national newspapers and long-established broadcasters. Candidate sources include (in alphabetical order): the BBC, The Guardian, The Independent, the Daily Mail, The Daily Mirror, The Observer, The Sun, The Telegraph, and The Times. There may follow a horizontal expansion (across the UK) or vertical (across the world). Whichever way is chosen, the intention is to include ‘alternative’ sources, which enhances their visibility and therefore their scrutiny.

The articles themselves can initially be confined to certain topics. One possibility is to choose those where opportunities for comments are sparse. However, these are likely to be the most controversial topics, the most difficult to handle well. A fledgling service should start with easier fare, build up experience and confidence, and then expand in a measured way to take in more challenging areas.

News Access

The system will use a database to store basic details of each news item: the publication (source), URL, title, date, author, type (bulletin, feature, etc.), keywords, comments status (open, moderated, email, closed, etc.) and access details. The design can follow conventions established by news aggregators sites and apps, of which there is a large choice. (Note that Reddit has news aggregator functionality, but it’s more general than that.)

There should be search and browse facilities to quickly find specific items and items on a particular topic or theme. Insights may be gleaned from news coverage as a whole. So, we can find answers to questions such as: What proportion of articles on climate change are open to comments?

Machine can use techniques in natural language processing (NLP) to carry out topic analysis and sentiment analysis.

Participation

There are many successful online communities, ranging from Meta’s offerings of Facebook groups and WhatsApp to Reddit and Stack Exchange. These last two are both distributed question-and-answer (Q&A) systems, but whereas Reddit encourages subjective contributions, Stack Exchange is not interested in opinions, but objective solutions. As with Stack Exchange's history, many started out as programmers’ hobbies and extra-curricular activities, but subsequently the user base has grown and diversified, with the subject coverage becoming encyclopaedic. The experience gained in these kinds of communities should inform service design, whilst I also wish to introduce a Buddhist perspective, as I believe this is rooted in a deep understanding of behaviour in human interactions.

Following general practice, discussion of current affairs are text-based and do not include audiovisual multimedia, only links to such content. This should be easier to manage.

User accounts

Should contributions require a user to be registered (with minimal personal data: first and last name for identification, a forum username and an email address)? Perhaps the option of anonymous comments might be supported, albeit with additional constraints (as per The Register’s Anonymous Coward).

Bots are not allowed (and don’t try to impersonate one and say “Did you come here because you are irritated? Come, come, elucidate your thoughts.”)

Guidelines based on the Buddha’s teaching on speech

The main guidance on contribution would be ‘Five Star Speech’, a direct appropriation of the Buddha’s teachings where one contributes:

  1. truthfully
  2. in a mind of good will
  3. expressed in a kind manner
  4. something relevant and beneficial to the recipient
  5. at the right time.

Vāca Sutta (On Speech), Anguttara Nikaya (Numerical Discourses of the Buddha) number 5.98, translated by Thanissaro Bhikkhu

These are not rules, but users are strongly encouraged to try to contribute along these lines as far as possible as it will enhance the overall experience and make it easier to adhere to the actual rules. These will be in the terms of service, which will be along existing lines and may borrow from those that already exist for social media.

Moderation

This is the main requirement of managing the service and for its basis, I quote again from the Buddha:

Intention, I tell you, is kamma. Intending, one does kamma by way of body, speech, & intellect.

Nibbedhika Sutta, Anguttara Nikaya (Numerical Discourses of the Buddha) number 6.63, translated by Thanissaro Bhikkhu

In Buddhism, actions in body, speech and mind, termed in Sanskrit karma, sow the seeds of happiness and unhappiness. When comments are laced with ill feeling and hatred; we refer to it as hateful speech. The words that result from hatefulness might be termed hate speechem> but need to be contextualised since the same words expressed in other contexts, with a benevolent mind, are entirely different in nature. This means that more effort is needed to ensure that contributions are appropriate, which we will address below.

Requiring users to register has some positive impact in terms of self-restraint, but is not sufficient. Skilful moderation is key to the viability of any such service, maintaining its civility. I envisage a combination of manual oversight supported by automated monitoring, with the adoption of some existing best practices, but with more focus on the nature of speech.

There are many facets to this, so, just as an example, monitoring should spot trolling patterns and then advise (with evidence) the poster to review their contributions.

Human moderators

Who should moderate? What should the criteria be? Essentially those who understand, uphold and encourage the principles of well-informed and kindly dialogue in the context of online discussions of current affairs.

The kind of scenario they would deal with would be a subscriber email or automated notification that one or more posts in a particular thread has crossed a ‘bad speech’ threshold. Their role is then to pay closer attention to that discussion, using their experience to facilitate it in a way that keeps it on topic and generally in accordance with 5-star speech. Mindful of the controversy concerning editorial practice at Wikipedia, expressed by its co-founder, Larry Sanger, for example, in an interview with UnHerd, I feel special care is needed to avoid and mitigate bias as far as possible.

So, the kind of person I would like to largely carry out this role is not someone with expertise or strong views on the topic, but rather someone who is a good listener and discerning in human interactions. A service that brings together people who are potentially at opposite ends needs very skilful support. So who is qualified for such facilitation? I have in mind experienced practitioners of Nonviolent Communication, a practice developed by Marshall Rosenberg to solve problems through communication rooted in empathy. Ideally, when problems arise, they enter into a dialogue with the poster, but it depends on resourcing.

Whilst they might not by themselves be able to determine the pertinence of an argument, the truth of a statement or validity of evidence put forward, they are trained in techniques to facilitate that process. Suppose that someone keeps insisting on an assertion, but complaints are received claiming that the assertion is false. What then? The original poster may be invited to clarify, but at this point some authority in the subject matter might be called upon to arbitrate. If no resolution is found, then what kind of closure is appropriate? Closing the thread? Or, as a general measure, limiting the number of posts to a given thread? Whilst wikis can descend into edit wars, in these fora, one can only edit one’s own posts. Some ‘hot topic’ or ‘contested’ flag might be raised.

There ought to be some kind of rewards system as well for those who have made substantial and highly regarded contributions. These might include badges and free newspaper subscriptions.

The use of automation to support moderation

We can enable the sites to be self-moderating to some degree through the use of devices such as a reputation system, commonly indicated by ‘likes’; in Stack Exchange, users largely determine this by indicating support for or approval of posts by others.

However, to aid efficiency and to improve coverage, automated moderation should be deployed, running in the background. Just as topic analysis and sentiment analysis can be used to give an overview, so it can also be used to discern problematic posts such as the trolling already mentioned. Algorithms can fulfil many assistive functions such as automatically grading topics for sensitivity, to give a quick overview, refined with human guidance.

At this point, a note about artificial intelligence (AI). The meaning of AI has narrowed to be, in many people’s minds, synonymous with machine learning, with the emphasis on machine capability. In contrast, I give centrality to the paradigm of Augmenting Human Intellect, as conceived by Douglas Engelbart in his report for Stanford Research Institute in 1962. This approach, as exemplified by his ‘Mother of all demos’, emphasizes the potential of machines to help us do things better and more efficiently.

The actual methods used should also be given careful consideration — I would prefer to avoid the use of ‘black box’ methods such as neural networks wherever possible, especially for reasons of transparency.

Management of Sources

The system should collate, store and readily make available metadata about the original articles, so that visitors to the site know their provenance. Ideally, each one should be archived in more than one way, including on the Wayback Machine at Internet Archive. The sources typically depend in turn on other sources, so there is scope for web crawlers, with many possible foci, such as exploring dependencies of thought.

Discussions, will bring up further sources. So, contributions should be mined also, building up a knowledge base available for analysis. This knowledge can be made available in various ways through the user interface and programmatically through APIs. For example, each article can have a tab that summarises the sources mentioned, perhaps with weightings, though that needs to be robust against gaming the system.

System Architecture

I’ve hardly given any thought to this as I’ve only been pondering what the service might look like. However, one aspect that particularly interests me is the potential for a distributed architecture. So, I would like to explore further Bluesky’s decentralised approach, as discussed in less technical terms in The Pragmatic Engineer.

Conclusion

So there it is. An idea, a tentative proposal, which, I hope, will give fresh impetus to commentary on current affairs, facilitating a fresh dialogue among and between people of differing views. A service for individuals, citizens of the world, that might even help in resurrecting ‘What the papers say‘, last seen at the BBC.

I’d like it to be turned into a business case. I’m not the best person to do this, but I can see how it might be of professional benefit to many parties: it should drive traffic to news sources and various professionals will find it a valuable repository for public opinion.

I’ll just add that an important consideration is how to maintain the service’s independence and protect it from undue influence (in the spirit of the declaration on freedom of expression). Policy statements are just one aspect. To increase its chances of longevity, the funding needs to be well spread and, when implemented, the moderation principles to be firmly maintained, true to the original intention.

Friday, October 07, 2022

ptworld.net renewed for the Sigala project

My first foray into Internet consultancy was in 1998, under the guise of PT Internet, but it was short-lived as I took up a full-time post a few months later.  However, wanting to maintain some online foothold, I bought the domain ptworld.net.  I naively assumed that a .net domain could denote an association with any kind of Internet services, whereas, at the time, it was intended for organisations involved in networking infrastructure or Internet Service Providers.  However, after a few years, with the proliferation of top-level domains, such a specific association faded away and I eventually got round to create a website, but sadly it became quite neglected until this year ...

ptworld.net home page (part) - depicting banner image (Doha's Corniche) and introductory text and links

After months of preparation, I’m pleased (and relieved) to have launched a renewed website, up-to-date and providing extensive coverage of what I’m about – especially in terms of research and software development, and my experience.  The site’s navigation reflects the interdisciplinary nature; hyperlinks are used liberally to allow ease of movement between the more theoretical research topics concerning Buddhist ethics to more practical perspectives such as cognitive interventions using thinking routines.  Regarding software development, I try to take a historical perspective when solving problems, which I incorporate also in my portfolio.   Alternatively, connections on a given themes may be explored via the tags or the use of the search facility.  

The banner image is a photograph I took whilst living in Doha.  It depicts the Corniche, with promotional slogans in English and Arabic.  The complete set (in English) is ‘Think’, ‘Realize’ and ‘Achieve’ — remember that Arabic is read from right to left.  Certainly, they have already achieved a great deal for a small state and the World Cup is arriving very soon.  It serves partly as a prompt to myself to persevere and keep the dream alive, as it were.  I’m not so good at self-promotion, but at this juncture the website is primarily an invitation to help nurture a viable business to support Sigala, a project that I have researched extensively.   The project needs to convincingly answer the question of why it’s a good investment.  How can the trusted network be leveraged to generate income ... ?

In terms of roadmap, there remains the need for a proof of concept.  Whilst I’ve established that Symfony is capable of delivering prototypes, I have found that consolidating all the project strands has been quite an undertaking and have not yet progressed this.  It may be better for someone else to do the coding, so with that in mind my next step will be to create mockups. 

Another reason for the delay has been a detour, which led to a mini project in itself to address the problem of delivering a faster and more secure public-facing website.  You may notice with the ptworld.net site, that URLs typically end in ‘index.html’, which means they are static pages, with no processing overhead from scripts or database queries to deliver the content.   All being well, you should notice that translates to faster page loads in your web browser.  The software responsible for delivering such pages – the result of this mini project – is MakeStaticSite, a set of Bash shell scripts released under the AGPL license, invoking popular GNU tools such as wget and rsync to generate and deploy with various options a static site.  It’s an age-old problem, but my focus has been on deploying live sites rather than creating archives, and is aimed especially at converting sites made in WordPress, though it works generally with any site accessible via http.   

Many developers consider WordPress retro, old or out of date, and prefer more modern approaches that are unconstrained by such a monolithic architecture and thus replete with possibilities.  But whether making WordPress headless or dispensing with WordPress altogether, it is acknowledged that it carries with it a significant technical overhead for development and maintenance.  I doubt that many individuals and small organisations are concerned about the constraints of themes; they will be happy enough if there’s a nice-looking result that clearly conveys the nature of their activities and will reach the intended audience. WordPress already satisfies their needs, so for them, at least, MakeStaticSite can allow this to continue whilst providing the additional benefit of delivering a safer and faster public-facing site. 

MakeStaticSite is already quite functional and I can see many possible improvements, but I need to move on as time is really limited.  If only I had a good and capable team at my disposal ... 




Saturday, February 19, 2022

Notes on Symfony 5: The Fast Track

Symfony 5: The Fast Track (front cover)

With the development phase for the Sigala project just starting, I’ve been looking to facilitate the swift delivery of a functional proof of concept using a mature PHP framework with many basic components provided in a well-structured way.   Enter Symfony, which I first heard about in around 2015 whilst taking baby steps in MVC by adopting Twig templates with my own interpretation of models and controllers.  Then, last autumn, I dipped my toes in the water with an accessible YouTube tutorial  delivered by Mohammed Kaabi for freeCodeCamp.org

Now, to really see what it’s capable of and to bring me up to speed, I’ve worked my way through version 5 of Symfony: The Fast Track, the official Symfony book by Fabien Potencier, its founder, released under a Creative Commons CC BY-NC-SA 4.0 license through the generosity of supporters.  The sentiments expressed in the book’s introduction fit well with my intentions,

The book describes the creation of an application, from scratch to production. ... Best practices won't be respected all the time. But we are going to touch on almost every aspect of a modern Symfony project.

While starting to work on this book, the very first thing I did was code the final application. I was impressed with the result and the velocity I was able to sustain while adding features, with very little effort.

That ability to add in features quickly is just what I’m looking for as I try things out, somewhat like working with a maquette.  That's the theory.

So, in this post, I share personal notes and views from following the book in coding from start to finish a conference guestbook.  I make reference to version 5.2  which was originally released in February 2021 and the book’s latest version until ten days ago (9 February 2022) when it was updated to support the latest Symfony releases (5.4 and 6.0, as of writing).  Furthermore, it looks like the book is becoming core to the project; the updates to keep it current are welcome, particularly for 5.4, which comes with long-term support.  However, in my case it was too late to switch.  Actually, from a quick glance, there don’t appear to be that many changes so far – there is helpfully more explanatory text, but as far as the code is concerned, it’s mainly a case of updating it to work in the latest versions.
 
I’ve selected a few themes from the many that the book introduces:


Afterwards, I offer some evaluation.

(I’ll refer to the author as ‘Fabien’, which seems to be the norm; I trust that he doesn’t mind.) 

Project Setup

There is a lot to set up.  It’s not until the creation of entity classes in chapter 8, that we have some PHP coding!  This may be routine for a professional full-stack developer, but for a novice or someone who is more narrowly focused on programming it may be off-putting.  I’ve heard it said that if you write a book about mathematics for the general public then you can expect the readership to halve every time you introduce a new equation.  I think something similar may apply in any tutorial-style work about software development for newcomers where various tools and services are required.

Work Environment

The checklist for this book (which Fabien has naturally automated with a symfony command!) presents quite a high bar.  After installing a few packages and libraries, I managed to ‘tick the box’ for all the required components, though I didn’t stick with Docker, as I shall explain later.  Also, I immediately diverged on a few of the recommended items.  The main one was the database engine – I switched from PostgreSQL, which I last used in 2008, to MariaDB, as that (or MySQL) is what I have been habitually using for my projects.  And I think of PHP and MySQL as forming a harmonious relationship ever since Monty Widenius collaborated with Rasmus Lerdorf in the 1990s

The first time Widenius met a PHP developer was when he saw Rasmus Lerdorf speak at an open-source conference. When Lerdorf explained how persistent connections were handled, "I thought it was unnecessary to open a new connection for each user," said Widenius, so I asked him "If I added some changes to MySQL would it help?" When the answer was yes, Widenius modified MySQL and the problem was solved on both ends.
 
Even so, I realized that I was taking a risk as there are differences in SQL syntax and Postgres has some features that MySQL lacks.  In the event, whilst I needed to tweak some SQL statements, I didn’t observe any lack of required functionality.

For the IDE, I used NetBeans, which is free and well-established; I’ve used it for most of my recent projects.  My main grumble is that the IDE is unusable when it carries out the initial background scanning of projects that include many third-party libraries (as is the case here), but the benefit is that it subsequently knows where they all are and how to access them.  Some configuration is needed to deal with other performance issues – such as adding these libraries, together with var/ folder to the search exemptions.  Once dealt with, I found NetBeans worked well, but I’m aware that some other IDEs, including Visual Studio Code, offer more fluent working with PHP.

I did not try Symfony Cloud – whilst its promotion is understandable, as an individual learner I'm not much motivated to use it, especially as the free trial period is only 7 days.  However, for a company employee, it may well be appropriate.  Fortunately, its omission doesn’t obstruct the book’s flow.  Among the other deployment options, Deployer appeals to me more, but again, not necessary at this stage.

Docker and MariaDB

My only prior experience of Docker was to evaluate some machine vision software by Oxford University’s Visual Geometry group, which used it as a distribution mechanism.   Their software required a number of specialist components, which would be quite an effort to install one by one.  However, in this instance, to use Docker, essentially a Linux VM for a single database instance with hardly any data, seemed overkill; I guess my view is that it's 'horses for courses.'

Despite my reservations, I went along with Docker and set up an instance of MariaDB.  Fortunately, Docker Compose is obliging:

$ ./bin/console make:docker:database

This asks which database engine to use, including MariaDB, and then creates the docker-compose.yaml file accordingly.

For Postgres, there is a nice command where environment variables are picked up, so there’s no need to remember credentials or port number, but for MySQL or MariaDB support is yet to be added,  So, I set up manually by modifying doctrine-compose.yaml:


version: '3.7' services: database: image: 'mariadb:latest' environment: MYSQL_ROOT_PASSWORD: password MYSQL_DATABASE: main ports: - '3306:53074'


With port forwarding established, I could duly administer the database from my machine as though it were local.  I then configured phpMyAdmin to allow Web admin, by adding a few lines to config.inc.php:

/** * Second server */ $i++; /* Authentication type */ $cfg['Servers'][$i]['auth_type'] = 'cookie'; /* Server parameters */ $cfg['Servers'][$i]['host'] = '127.0.0.1'; $cfg['Servers'][$i]['port'] = '53074'; $cfg['Servers'][$i]['compress'] = false; $cfg['Servers'][$i]['AllowNoPassword'] = false;

The port, which is randomized, needs to be set each time Docker is restarted.  

Abandonment of Docker

Whilst on Chapter 14, I was no longer able to connect to the database instance on the Docker Container because the port forwarding changed and seemed to only allow internal access.  After trying a few things, I gave up trying to fix the problem, launched a CLI terminal from the Docker Dashboard and dumped the database to a file, 

# mysql -u root -p -h 127.0.0.1 main > /tmp/main.sql

I then copied the file from the container (guestbook_database_1) onto the local filesystem:

$ docker cp guestbook_database_1:/tmp/main.sql .

Then, I set up a fresh database locally and imported the dump.  Now that Docker files wouldn’t be used, I edited env.local to include (on one line):

DATABASE_URL="mysql://fasttrack:password@127.0.0.1:3306/ guestbook?serverVersion=10.6.4-MariaDB - Homebrew"
  
From that point on, I happily continued with the local MariaDB instance.

SQL Tweaks

With much of the SQL being created through the Symfony MakerBundle, I didn’t have to make many changes to the SQL.  However, there were usually tweaks to be made wherever Fabien had manually modified [Postgres] SQL after running make:migration.  

13.2: Adding Slugs to Conferences

Manual changes to deal with the issue of existing entries getting a null value.

    $this->addSql('ALTER TABLE conference ADD slug VARCHAR(255)');
    $this->addSql("UPDATE conference SET slug=CONCAT(LOWER(city), '-', year)");
    $this->addSql('ALTER TABLE conference MODIFY slug VARCHAR(255) NOT NULL');

15.3 Creating an Admin

Rather than run a fairly long query at the command line, I used phpMyAdmin.  Log in and select the guestbook database and then the admin table.  Select the [SQL] tab along the top and then [INSERT] tab under the text area window.  It automatically generates a sample that includes id, but since it is auto-incrementing, we don’t need to define it.  Thus, we can remove that field and copy & paste in our encoded password that was generated from the command line, something like:

INSERT INTO `admin`(`username`, `roles`, `password`) VALUES ('admin','["ROLE_ADMIN"]','$argon2id$v=19$m=65536,t=4,p=<hash>')

18.1 Flagging Comments 

This section introduces a state for comments: submitted, spam, and published and adds the state property to the Comment class.  For MySQL, the SQL for setting all comments to have the default published:

+ $this->addSql(ALTER TABLE comment MODIFY state VARCHAR(255) NOT NULL ');

18.5 Going Async for Real 

This is just an observation, no SQL code changes.   Fabien remarks: “Behind the scenes, Symfony uses the PostgreSQL builtin, performant, scalable, and transactional pub/sub system (LISTEN/NOTIFY).”  That sounds like a sales pitch, perhaps USP, for PostgreSQL; a quick lookup seems to show that MariaDB doesn’t have an equivalent to LISTEN/NOTIFY.   I don’t really know the significance.  All I can say is that I didn’t notice any issues in database transactions (messages seemed to get passed between application and database as expected).

Code

I’m not well-qualified to judge here (I’ve never done code reviews for others), but Fabien's code seems polished, concise and precise.  Apart from the book’s narrative, it’s generally uncommented.  So, as a relative newcomer to Symfony, I found myself having to look up various sources for elucidation.  Fortunately, the documentation provides excellent coverage and the video tutorials provide very helpful explanation.

I encountered some issues as I went along, some of which were probably due to my not following instructions correctly, but I’ll mention the main ones I had in case others encounter the same problems.

Dependency Issues

I had a few of these along the way, both in composer and yarn.  The first arose when installing the profiler:

$ symfony composer req profiler --dev Warning from https://flex.symfony.com/versions.json: You are using an outdated version of Flex, please run: composer update symfony/flex Using version ^1.0 for symfony/profiler-pack

To solve this, I ended up modifying composer.json, replacing “5.2.*” with:

"require-dev": { "symfony/stopwatch": ">=5.2", "symfony/web-profiler-bundle": "5.2.*" },

(I should have compared by checking out from the official repo.)

However, I wasn’t clear about what I was doing.  So, it would be helpful to have some guidance on how to deal with these issues – I’m sure that with experience they are resolved much more easily.

Not null constraint on Creating Comments

In Chapter 9, just before the section ‘Customizing EasyAdmin’, the text invites the reader to add some comments.  But it’s not yet possible because there’s no reference to conferences in the user interface.  When trying to submit a comment by hitting the [Create] button we get an error: 

An exception occurred while executing a query: SQLSTATE[23000]: Integrity constraint violation: 1048 Column 'conference_id' cannot be null.

From the profiler ‘Logs’ tab, we can see that Doctrine tried to insert a null value into this field, but a not null condition is specified in the Comment entity:

/**
  * @ORM\ManyToOne(targetEntity=Conference::class, inversedBy="comments")
  * @ORM\JoinColumn(nullable=false)
  */

    private $conference;

This condition was originally set when we ran make:entity for the conference field.

To take account of the Conference, we need some mechanism to link the comment with the relevant conference and that should be made by the person writing the comment.   The configureFields() method in CommentCrudController.php is amenable and EasyAdmin can list the conferences in a drop-down.  The following solution prepends the conference column to the configureFields() method:

$fields = parent::configureFields($pageName);
        array_unshift($fields, AssociationField::new('conference'));
        return $fields;
)
        
The issue was already reported and this is just a variant of the solution offered there, though I don’t yet see it in the latest edition of the book.

Coding Variations

In a few cases, I used slightly different code to achieve the same result. For example, in Chapter 10.3: Using Twig in a Controller, I drew on Kaabi’s tutorial for a simpler rendering of the template in the index() function.  In the case of extending AbstractController, we don’t need to explicitly reference Twig.  We can just refer to the instance of the conferenceRepository, using a technique explained in the documentation.

class ConferenceController extends AbstractController {

    #[Route('/', name: 'homepage')]
    public function index(ConferenceRepository $conferenceRepository): Response {
        return new Response($this->render('conference/index.html.twig', [
                    'conferences' => $conferenceRepository->findAll(),
        ])->getContent());
    }

(The getContent() method just returns the content of a response, an alternative to using RenderView().)

Workflows: State Machines

The treatment of workflows, particularly the section on their description, sparked my interest in its direct connection with state machines and their formal representation in Petri Nets.  When using formal description techniques, we can take advantage of mathematical methods that add rigour to tasks of verification (that processes are behaving correctly) and validation (that we are designing the right processes).  In my PhD, I used process algebra (LOTOS) and temporal logic as the formalisms to study safety-critical systems.  My present interest is in [human] safety properties in the design of social media – not just protection from physical harm, but mental harm.  Taking a Buddhist model, this is protection across the khandhas – form (body) together with feelings, perceptions, mental formations and consciousness (mind).  It's a topic that requires a separate blog post.

User Notifications

On a practical note, workflows underpin user notifications, where we have the one instance where (in 25.5) the reader is invited to write their own code.  As the last task of the main website construction, users are to be notified when their submission is approved. There are probably many ways to achieve this.  I’ll share an attempt here using a workflow-oriented method.

This first step is to create a template with a basic acceptance message, comment_accepted.html.twig

{% extends '@email/default/notification/body.html.twig' %}

{% block content %}
    <p>Dear {{ comment.author }},</p>
    <p>Your comment (copied below) has been accepted!</p>
    <p>
        {{ comment.text }}
    </p>
    <p>Thank you for your contribution.</p>
{% endblock %}

Thus far, notifications have been used only for admins.  The message is sent via:

$this->notifier->send(new CommentReviewNotification($comment), ...$this->notifier->getAdminRecipients());

The second parameter of the send() method has to be of type Symfony\Component\Notifier\Recipient\RecipientInterface

We have to interact with a class that implements RecipientInterface – directly or as an extension, etc., or possibly create such an implementation or extension ourselves.  Symfony’s documentation on Notifiers helpfully gives us the answer in the section on creating & sending notifications.

In InvoiceController.php, the key line is:

   $notifier->send($notification, $recipient);

So, in CommentMessageHandler.php we include the Recipient class, 

use Symfony\Component\Notifier\Recipient\Recipient;

And instead of ...$this->notifier->getAdminRecipients() use $comment->getEmail() for the email address submitted in the comment.  Then we might we anticipate defining a new class, CommentAcceptedNotification, to return a different message output.   

However, instead of having separate classes for different comment-related notifications, we can fold the code into one, by augmenting CommentReviewNotification with further methods and properties.

So, immediately after 

$this->workflow->apply($comment,'optimize');

insert:

$this->notifier->send(new CommentAcceptedNotification($comment), new Recipient($comment->getEmail()));

Furthermore, we can determine what notifications to send based on the workflow, which has been defined to support comments.  We do this by reviewing config/packages/workflow.yaml and its corresponding state transition diagram:


State transition diagram for complete workflow (Graphviz output)


We can choose rules based on places and/or transitions.  The book has based rules only on transitions, but here I think it’s better to start with places, which are unique, whereas in general transitions might not be so.  

For this exercise, we just want to add the case where the place (state) = published.

The getters and setters are already there in App\Entity\Comment.  

/**
  * @ORM\Column(type="string", length=255, options={"default": "submitted"})
  */

    private $state = 'submitted';
    
and

    public function getState(): ?string
    {
        return $this->state;
    }

    public function setState(?string $state): self
    {
        $this->state = $state;
        return $this;
    }

So, we update CommentReviewNotification to determine the current place with

$place = $this->comment->getState();

And add a map from states to templates.  Thus we ended up with the following for the  class CommentReviewNotification


class CommentReviewNotification extends Notification implements EmailNotificationInterface
{
    private $comment;
    private $place;
    public const NOTIFICATION_TEMPLATE_MAP = [
            'potential_spam' => 'emails/comment_notification.html.twig',
            'ham' => 'emails/comment_notification.html.twig',
            'published' => 'emails/comment_accepted.html.twig'
        ];

    public function __construct(Comment $comment)
    {
        $this->comment = $comment;

        parent::__construct('New comment posted');
    }

    public function asEmailMessage(EmailRecipientInterface $recipient, string $transport = null): ?EmailMessage
    {
        $place = $this->comment->getState();
        if (isset(self::NOTIFICATION_TEMPLATE_MAP[$place])) {
            $notification_template = self::NOTIFICATION_TEMPLATE_MAP[$place];
        } else {
            $notification_template = 'emails/comment_notification.html.twig'// should be a different template to cover other states (or we're lost!)
        }
        
        $message = EmailMessage::fromNotification($this, $recipient, $transport);
        $message->getMessage()
                ->htmlTemplate($notification_template)
                ->context(['comment' => $this->comment])
                ->importance(Notification::IMPORTANCE_LOW)
        ;
        return $message;
    }

 ... 

}

Whilst quite a neat solution, I think it would be more consistent to put this logic in CommentMessageHandler.php, along with the conditionals for transitions, where, again, I think using places might be better.

User Experience

In more recent projects, I’ve observed a continual shift towards the frontend, to user experience.  Mindful of its importance, what do we observe from this book?  Not so much.  The treatment is a bit lightweight.

User Interface

Whereas Kaabi introduced Bootstrap almost immediately after outputting just the main heading in a template, here, it’s not until chapter 22, after coding much of the application logic, that Fabien shows how to style the user interface.  There’s no discussion of how the templates have been designed – you just download them and are encouraged to “Have a look at the templates, you might learn a trick or two about Twig.”  That’s it!

For Symfony, I can see why it’s not a high priority – what’s of greater relevance for a framework like this is the way integration is performed in Webpack.  I sympathise with this view, but I think the book could be quite a bit more popular if greater attention were given to this area.  And the information I seek might be readily available – I would have particularly welcomed more pointers to illustrate the work of the UX initiative  (see also its rationale).  As it stands, it’s probably where I feel there are the most question marks.  

At least, with the SPA, styling was introduced before data was fetched. 

Incidentally, in the text encountered few display issues, e.g. in creating the SPA main template, my browser does not render the code listing for src/app.js correctly.  The <div> tag around ‘Hello world!’ has disappeared and the code to render should be:

render(<App />, document.getElementById('app'));

Internationalization (i18n)

I’m glad this section was included, not least because it helped clarify a standard approach to delivering localized content.  Previously, I using Gettext for a bilingual (English LTR/Arabic RTL) collections intranet for the Qatar Museums Authority, with the ability to toggle the locale on any page.  Most of the work for the translations was carried out by a Syrian colleague, who translated every inventory field that could be displayed. 

In the conference guestbook, I added a Thai version (notice the months notation in the date):



As it stands, entering translations for RTL languages will not change the direction of page contents.   To quickly (and lazily) see the effect, I generated a translation file for Arabic with the following additional translation unit:

      <trans-unit id="7654321" resname="LTR">
        <source>LTR</source>
        <target>RTL</target>
      </trans-unit>

Then edited the <html> tag in base.html.twig:

<html lang="{{ app.request.locale }}" dir="{{ 'LTR'|trans }}">

The result is:


It makes the key adjustment for the page as a whole, but some blocks are still styled left-to-right.  

It would also be useful to indicate how to implement i18n on the SPA. 

One further remark, about providing the book itself in different languages.  Up until this version, 5.2, it was available in more than a dozen languages, including Arabic.  It's evidently been a huge amount of work and going forward will be hard to maintain and so far I see only European languages for the current versions.  But perhaps automated translation is now mature enough to facilitate?  Whereas 10-15 years ago, the results could be a bit comical, they have in recent years improved considerably.  Symfony provides integration with third-party providers (perhaps DeepL be added to the list?) which can offer a very useful starting point.  Furthermore, the language of programming is probably more consistent than the average text.

Pondering Deployment

When to use?  At first glance, it seems natural to just deliver website.com/fr/ to French-language speakers or website.com/es-AR/ to inhabitants of Argentina.  However, it is not always appropriate to view a website in your own locale if, for example, you are learning another language and want to read local news in other countries.  Many people are of mixed ethnicity and multilingual – it is quite common for children of immigrant families to speak the language of the host country at school, but their mother tongue at home.  This suggests there should be flexibility in switching between languages.  

This seems to be tacitly recognised on the Web when you first visit a site and get a popup saying ‘You are viewing this page in <locale>, but [based on your IP address] you are viewing from <country Y>.  Do you want to view in your own locale?  More generally, I’d like to push beyond the mechanical translations to cultural translations – how to more authentically support online the process of introducing oneself, which can vary considerably between cultures?   

Evaluation

The book is well-written and conveys convincingly the tremendous achievement of the Symfony project.  It’s really a showcase rather than a tutorial, yet is very instructive.  With its lean and concise descriptions, the book covers an extraordinary amount of ground – the order in which new material is introduced builds on previous matter and leaves nothing to waste. 

It took me weeks to work my way through, but a lot of that is down to my lack of familiarity with various components that many will already be using on a regular basis, including third-party services such as Akismet and Slack (I think using the API unlocked the 30-day trial of premium features).  I feel it’s worth the effort, especially as I’ve learnt a lot about the framework’s potential with useful pointers about implementation, which is the main reason I took on the book in the first place.  It has considerable capacity to support the development of complex applications in an efficient and manageable way, leaving me with the impression that this really is a robust platform.

Somewhat unusually for an introductory book to a language or framework, it provides significant coverage of topics relating to production systems – their monitoring and efficient delivery.  I’m not yet at the stage where I need to concern myself with these matters, but this ‘systems thinking’ shows foresight and experience of running systems for real; it’s very welcome.  Furthermore, they are placed in the text in such a way that much of the material can be skipped (I have not yet read through the last three chapters).  Whilst I originally decided to use Symfony for its provision of many components, I have been greatly impressed by what I’ve seen, so I think it is a serious candidate for delivering the software at every stage, right through to production and ‘business as usual’.

Whilst I followed the instructions from the Website, I purchased a printed (PDF) version to support the work (and I did enjoy the family artwork!).  One suggestion.  In return for payment, it would have been nice to have added a sample month’s subscription to symfonycasts.com, to support further in-depth exploration.

Fabien continues to be very active in Symfony’s further development and promotion (see, for example, his responses in a live Q&A from Feb.2019.  The community supporting the project also appears well-engaged, which all bodes well for the future.