This site uses cookies so that I can improve the user experience. You can read more here.

This article was created with legacy styles and may look a little strange!

My learnings from HalfStack's Conference

ACCESSIBILITYPOSENETRESPONSIVE-DESIGN

Some notes, key takeaways and hacky prototypes that I created during my first ever virutal conference! The conference was focused on UI-centric JavaScript and web development.

LEARNINGS FROM HALFSTACK CONF 2020

Building Motion Controlled Web Applications

Charlie Gerard - Senior (Front-End Engineer at Netlify) gave an awesome talk on how we can use some great new HCI devices like the "Myo Armband" to change the way we interact with our browsers.

Common HCI

  • Keyboards
  • Touch
  • Swipe
  • Voice

Incoming HCI

  • Gyroscope
  • Leap Motion
  • Myo Armband

JS Camera Libraries

There are a crazy number of JS libraries that make use of a webcam. If you're working in ReactJS theres an awesome npm package called "react-webcam" thats great for getting started. We explored some libraries that can make use of webcam data in detail below.

Face Mesh

Uses key-points to get X,Y and Z axis. Uses a model to make predictions on the video source about the presence of the face.  Returns a list of different parts of the face, bounding box of the face and mesh data. (Lots of arrays!)  Built POC app called face-scroll that allows you to scroll up and down the page using your head

PoseNet

Pose detection. recognises body shapes. Pulls in video feed from client - can disable audio. Uses a detect function to guess pose based off of identified body shapes. Has a confidence threshold  Made beatsaber clone in the browser.

Hand Pose

Hands only! Recognises key-points of the hand and has Z axis. Can recognise when fingers are folded. Syntax is similar to PoseNet.

Putting It Into Practise

I integrated pose recognition into my site's presentation clicker to create touch free slide navigation for presenters using just their gestures. This prototype is built using socket.io, posenet and one of my Gatsbyjs boilerplates which you can find here.

Limitations

  • All of this is super new - Google released in last couple of months so we should see more improvements in coming years. Max frame rate is 40fps right now. Smooth animations are normally around 60fps.

  • Its Heavy - Library is around 12mb for handpose. This of course, affects site performance.

  • Accuracy - Laptop webcams and not depth cameras and may not be super accurate. Lighting is super important to improve accuracy.

  • Privacy - Constantly tracking the user while recording video - concerns over privacy need to be considered.

  • UX - This is a new way to interact with the web and as such would need to be learned - but people would be excited and would probably adopt quickly.

Key Takeaways

  • These tools are really useful to help improve the accessibility of the web.

    We could in the future have a "motion-mode" like "dark-mode" on our websites. This could be a toggle feature that when on allows the user to navigate a website with any pose, gesture, facial expression.

  • We could use it to improve our health and fitness.

    We could create websites that encourage us to exercise or help us improve out yoga positions.

  • We could use it beyond the browser

    JS is not just for websites in the browser but also the internet of things. Imagine the possibilities?

"HCI is broad so start small - focus on one piece of the puzzle. Use one library or device and have fun!"

Responsive Design in 2020

Kilian Valkhof (Makes Polypane) gave a talk on the history of responsive design and whats coming up in the future with media queries.

Responsive Design

The way a website adapts to different screen-sizes by reflowing and repositioning content as the available space allows. Responsive design means:

  • Mobile first
  • Media queries width in ems
  • Base font size
  • Allow resizing
  • Scrolling is fine - no fold

Cheap Dark Mode

@media (prefers-color-scheme: dark) {
  :root {
    background: #111;
    filter: invert(1) hue-rotate(180deg);
  }
  img, video {
    filter: invert(1) hue-rotate(180deg);
  }
}

prefers-reduced-motion

Users can indicate they want to see less happening on the screen. Reasons they may want to do this may include motion sickness or maybe they don't want to wait for your animations to finish. This doesnt mean you can't use any motion, just be mindful.

prefers-reduced-transparency

Users can indicate they want to see things on solid colours. Reasons they may want to do this may include visual impairments but it can also help people who struggle to focus. This feature is not yet widely supported.

Network conditions: navigator.connection

We can use navigator.connection to determine whether a user would like to save their mobile data but also their effective network conncetion speed. You can see a simplified version of how this might work next to this!

Testing Network Conditions

if(navigator.connection.saveData 
|| navigator.connection.effectiveType 
=='slow-2g') {
 preloadVide = false;
}

Putting It Into Practise

The div next to this section has been set up using a media query to ensure that it always sticks to the users dark-mode preferance. You can use the dark-mode toggle at the top of the screen to switch it. It also detects network connection strength.

Your effective network type: Not supported by your browser!

Onto the next one!

Internationalisation and Data Visulisation

Naomi Meyer (SDE at Adobe) gave a cool talk on internationalisation and data visulisation in a COVID world!

"Data visulisations are powerful international communication tools to effectively communicate a storyand share critical information across the world."

Formatting a graph for the world:

  • Format integers correctly across each distinct local

    Different locals can have different formats for displaying numbers correctly.

  • Think about colours

    Different colours can have different cultural and regional signifigance so keep it in mind!

  • Dates and Times

    Dates are often overlooked when formatting.

  • Don't assume left to right

    There are many countries out there where people read right to left and its important to consider this when looking at these locales.

Culture

Culture is deeply rooted in our thinking patterns. Culture can effect how are uses interact and benefit from digital experiences. Internationalisation goes beyond language. By acknowledging cultural and social differences we're creating with innovation.

Touch screens were intially built for accessibility but now we are all reaping the benefits.

Reusable Formatter Instance

const reusableFormatterInstance  = (locale, number) => {
  return new Intl.NumberFormat(locale).format(number);
};

reusableFormatterInstance('en', 123456789123456789);

Integers

Integers are not the same across the world!

English --> 123,457 | Tamil --> 1,23,457

Further Reading

Putting It Into Practise

The div next to this section contains a demo of using internationalisation to format a string based on various locales. It uses compact notation to ensure it will fit the space.

Original Number: 12500100

English: 13M

German: 13 Mio.

Hindi: 1.3 क॰

Russian: 13 млн

Korean: 1250만

Tamil: 13மி

...

...

...

...

Mailbox

Join My Newsletter

Want to know when I post something new? For the latest articles and projects straight to your inbox, subscribe to my newsletter.