Database Endemes

Even though endemes are best used at level 3, looking through my database, I have noticed that there are things which can be handled by endemes. Here are some simple uses of endemes in a database. Databases commonly contain columns like:

  • status/enabled/disabled/active/inactive etc. etc.
  • flags/bits/indicators
  • type/category/class
  • priorities/importance/severity
  • modes
  • actions
  • access/permissions
  • options

With more complex frameworks, endemes can be used to store:

  • user fields etc.
  • simple field values
  • data mining results

Information Migration

Low level information is best stored using endemes. Endeme migration is an interesting situation. Part of the idea of information programming is getting it right (rather than getting it perfect). So your endeme sets will evolve as you change your endeme sets from time to time. But what about the endemes developed using a previous version of the set?

Four Solutions to the Endeme Migration Challenge

I can think of four possibilities for handling this situation in which endeme sets change over time in a production system.

  1. Migrate each of the old endemes to the new set, possibly using an endeme proxy translation for the migration.
  2. Allow the old endemes to degrade gracefully or perhaps less than gracefully.
  3. Create a new set and an old set and change the set for the old endemes to the old set (or just some of them) and change the code to use the correct set as needed.
  4. Throw the system away perhaps because it was not built architecturally or info-architecturally right

Certain kinds of endemes/sets will degrade gracefully

  • Small changes to sets will degrade more gracefully, larger changes may have to be migrated.
  • Since endemes are a form of qualitative information, qualitative factors are important. For example if a characteristic letter gains a radically different very important meaning, the degradation will not be graceful. Migrate in this case.
  • Endemes that control software may not degrade gracefully, endemes that characterize resources are much more likely to degrade gracefully. What matters most is how they are used.
  • The importance of what is done with the endeme set also weighs in.

The Process of Migrating Information

Migrating can be used for writing proxy endemes between the old and the new version of the endeme set. A translator endeme can be devised for many old to new set conversions. In fact translator endemes can be used for far more than just migrations.

Staffing a Full Stack Project

If you are staffing a full stack project you will need about 10 people, or at least have people who can together wear about 10 hats.

You will need a database developer, an object oriented developer, an information oriented developer, a knowledge representation expert, and artificial intelligence developer, and computational creativity developer, a front end developer, and network integration developer, a project lead, and a level integration specialist. You may also need a general IT person to handle infrastructure and a business analyst to handle contracts.

The integration specialist may be the most important person. They will need to help all of the other people develop so that each level is developed to be used by the one above, and so that each piece of work gets assigned to the right person and so that the stack built compiles and links together and continually works.

The lead person, works closely with the integration specialist to identify proper assignments, and works with stakeholders. The primary job of the lead person is to make sure that the stakeholders are aware of what a full stack program can do, and don’t ignore out the value provided by information and artificial intelligence layers. Since this stack is so far outside the mainstream, the lead person ends up being trainer of customers and stake holder to help them break out of their data oriented thinking and see what they can actually get for their money.

Each person needs to work closely with team members responsible for the layers above and below them.

Use Separation of Concerns with Information vs Data as well as Presentation vs View

Separation of concerns consists of separating one kind of code from another into different sets of code so that when one concern is being worked on, it has less of an effect on the other sets of code.

Examples of Separation of Concerns

Here are four separations of concern, three are well known, the fourth is proposed:

  1. [Presentation/UI]     and [Business logic/Model]
  2. [Business logic/Model] and [data storage/RDB]
  3. [Aspect orientation]  and [Functionality]
  4. [Information]         and [Data]

When applying separation of concerns to information concerns vs data concerns, how do you implement this? How do you implement separations of concerns regarding the information vs data distinction? What is your definition of information (vs data)?

Definition of Information vs Data

First of all, data and information are different in that information is data that has meaning. Data is data with little meaning. One way to spot the difference in real life is that usually, data comes in large quantities, whereas information is likely to be in much smaller quantities.

Information Oriented vs Data Oriented Programming

Within the realm of programming, data oriented programming works with data within a fixed context, and information oriented programming processes the context itself.  Fixed data oriented programming contexts can be things such as structs, classes, xml schemas, tables, and primitive variables and methods in a program. Each of these provides a fixed context for they data they contain. With information oriented programming, the context becomes variable. Members and portions of other fixed context listed above become variably defined fields using techniques to define the meaning of each field. An example of this is when a list of fields are each defined by an an endeme. Another example happens with knowledge representation through ontologies in which the meaning of each field is defined by relating it in various ways to other fields.

Implementing the Data vs Information Separation of Concerns

To implement the separation of concerns, the code used to define the field meanings and the code used to implement the meanings is moved into a separate section of the program from where the main application is kept. This is usually a separation of the middle tier in horizontally architected programs into two pieces. In a vertical architecture, each level is separated into framework and application code and the field definitions and meaning implementation are collected into the application code. I’m still in the blue sky phase for some of this stuff, so my opinion on separation of concerns may change over time. This is how I see it now.

Separation of Presentation vs Logic Concerns

Separation of data vs information concerns has a neat side effect. As I have been working with my test bed, I have also noticed that separation of concerns – data vs information enforces separation of model vs view (logic vs presentation) concerts .

The Biggest Challenge

The biggest challenge in implementing separation of concerns is being able to identify the difference between data and information and being able to identify which tasks, work, and areas of functionality belong to which layer.

Information Is Scary

information is scary
 - if it is true it may tell you things you don't want to know
 - it is not under control
 - users define the format, filter, and fields
 - results are not precise
 - results are powerful - revealing
 - it is exciting, warning, guiding

Comparing the difference between information and information technology

Information Programming        Information Technology        Information Workarounds
-----------------------        ----------------------        -----------------------
  business rules             - logical business rules        business rule engines
  (concept based search)     - field based search            keyword search
- HCI/CHI/UX                   console apps & debuggers      RDMS & BI tools
- usability                    programmer think              Phd think
  profiles                   - totals                        graphics
  interpreters               - strong typing, classes        cleansing data
  Traced UI's                  integration programming     - Untraced UI's
  dashboards                 - field and screen UI's         reports
  complex data sets          - large data sets               BI
  (characterizations)        - patterns                      ontologies
  customizable UI            - specified UI                  giant tool UI
  conceptual programming     - modern programming            special language programming
  code reuse                 - ORMs                          WORN
  MPV                        - MVC                           MVVM
  (soft exceptions)          - hard exceptions               corrupt results
  reveal anomalies           - sanitize anomalies            ignore anomalies
  generators                 - factories                     computational creativity
  content interpreters       - CMSs, Sharepoint              semantic webs
  (trickling, managed threads) - multi-threading, events     <undeveloped>
  (information awareness)    - Networks, services            semantic nets
  (endemes)                    structures                    freetext
  inherent & vector graphics - ascii & vector graphics     - ascii & raster graphics
<undeveloped>                  FTP, Protocols              - Email
<undeveloped>                - API's                         screen scraping
  information constructs     - Artificial intelligence       Neural Networks
  dynamic queries              ad hoc queries              - saved queries, sp's
  wikis, attention management  forums                      - social media
  writing programs (& using) - writing and using programs    using programs (& writing)

multi-threading information means compiling information from various
services over time with real-time modification of information (trickling) ()

The Uses of Information

Here are some things information oriented systems can be used for in business:

  • more business flexibility
  • easier mergerability
  • new opportunities
  • reduced costs
  • more competitiveness
  • real time metrics
  • transaction stream mapping
  • dashboards and dashboard API’s
  • business analytics
  • risk management
  • decisions
  • deliberate continuous improvement
  • structured qualitative metrics
  • resource management
  • user experience enhancement

An Agglomeration Around a Broken Core

Sometimes I find myself working with a set of software that is essentially an agglomeration around a broken core. The architecture is inadequate to sustain expansion, improvement, or long term survival into the future. For whatever reason the initial system – its initial core architecture is not right.

Various things get added to the broken core to handle specific problems with it and expand it a little into new functionality. These things agglomerate over time around the core system. so their systems end up being agglomerations around a broken core. This seems like some sort of anti-pattern. Perhaps there is and anti-pattern called “Agglomeration around a broken core”.

Software development has become an agglomeration around a broken core.

Computer science itself seems to be an agglomeration around a broken core. The symptoms are more and more systems to handle the complexities of software including lots of SOAP and services, and interfaces and contracts, frameworks and patterns and mounds of code to handle fields/members. This is a nice list of agglomerations.

These agglomerations are mostly ways to try to deal with the broken core of computer science. Not to say that they are useless. They do have appropriate uses. Mostly they are to wrap the broken core so programs work better.

The Ultimate Cause of the Problem

The broken core is that we are handling fields in a way that is not always correct. Sometimes we want hard coded fields for performance and precision reasons. Sometimes it is better to have soft coded fields, however computer science has fallen down terribly in the area of supplying technology support, tools and theoretical basis for soft coded fields.

The Solution to the Problem is in Level 3

The solution is to build level 3 so that it handles fields in a soft way.
Level 2 will always be better written with the hard coded fields.
Now we need to develop level 3. We can use endeme powered soft coded fields with each field identified by and endeme rather than entirely identified by the class or structure or interface that it is in.

Because computer science is not a single system, different technologies can displace others over time by being implemented in a few and then more and more places over time. Eventually the broken core can be replaced through the life/death cycle of individual projects and systems.

As long as the languages we have allow me to do this I can contribute to this. But I have to get my working core working first, which means getting the soft code fields of level 3 to work properly and then start implementing systems using this. That is my solution.