User Interface Software
User Interface Software CS 4470
Popular in Course
Popular in ComputerScienence
This 0 page Class Notes was uploaded by Alayna Veum on Monday November 2, 2015. The Class Notes belongs to CS 4470 at Georgia Institute of Technology - Main Campus taught by Staff in Fall. Since its upload, it has received 8 views. For similar materials see /class/234123/cs-4470-georgia-institute-of-technology-main-campus in ComputerScienence at Georgia Institute of Technology - Main Campus.
Reviews for User Interface Software
Report this Material
What is Karma?
Karma is the currency of StudySoup.
You can buy or earn more Karma at anytime and redeem it for class notes, study guides, flashcards, and more!
Date Created: 11/02/15
015 0599 COOS OCCCQ Damage Management amp Layout Georgia Tech E Tech Georgia My Damage management 0 Need to keep track of parts of the screen that need update 0 interactor has changed appearance moved appeared disappeared etc 0 clone by decaring damage 0 each object responsible for telling system when part of its appearance needs update Georgia Tech Damage management 0 Example in Swing done via a call to repaint G takes a rectangle parameter E Adds the speci ed region to the RepaintManager s dirty list 0 list of regions that need to be redrawn 0 RepaintManager schedules repaints for later can collapse multiple dirty regions into a few larger ones to optimize Q When scheduled repaint comes up RepaintManager calls component s paintlmmediately methodwhich calls paintComponent paintChildren paintBorders 0 You generally never want to call this yourself 0 Generally seldom need to work with RepaintManager directly Georgia Tech Damage Management 0 Can optimize somewhat 0 Multiple rectangles of damage 0 Knowing about opaque objects 0 But typically not worth the effort Damage Management in Swing JComponent Georgia Tech 4 RepaintManager n Georgia My Tech processing cycle I Typical overal loop forever wait for event then dispatch it gtcauses actions to be invoked andor update interactor state typically causes damage if damagedsomewhere layout Layout 0 Deciding size and placement of every object Q easiest version static layout 0 objects don t move or change size 0 easy but very limiting hard to do dynamic content 0 only good enough for simplest cases Georgia Tech l Dynamic layout 0 Change layout on the fly to reflect the current situation 0 Need to do layout before redraw 0 Can t be done eg in paintComponent Q Why Georgia Tech l Georgia Tech Dynamic layout 0 Change layout on the fly to reflect the current situation 0 Need to do layout before redraw 0 Can t be done eg in paintComponent 9 Because you have to draw in strict order but layout esp position may depend on sizeposition of things not in order drawn after you Tech Georgia My Layout in Swing 0 invalidate method 3 Called on a container to indicate that its children need to be laid out 0 Called on a component to indicate that something about it has changed that may change the overall layout change in size for example validate method G Starts the process that makes an invalid layout validrecomputes sizes and positions to get correct layout 10 Tech Georgia My Issues with Swing validation 0 invalidate is often called automatically 1 eg in response to changes to components state but not always 9 eg if a JButton s font or label changes no automatic call to invalidate 0 Mark the button as changed by calling invalidate on it 9 Tell the container to redo layout by calling validate on it 0 In older versions of Swing you had to do this by hand Newer versions post 2 add a shortcut revalidate 1 lnvalidates the component you call it on O Begins the process of validating the layout starting from the appropriate parent container Validation also uses the RepaintManager LayoutValidation in Swing JComponent Georgia Tech 4 RepaintManager 39f In 12 Georgia Tech Layout with containers 0 Containers parent components can control sizeposition ofcthren 0 example rows amp columns 0 Two basic strategies 0 Topdown AKA outsidein 0 Bottomup AKA insideout Georgia Tech Topdown or outsidein layout 0 Parent determines layout of children 0 Typically used for position but sometimes size 0 Example l 14 Georgia Tech Topdown or outsidein layout 0 Parent determines layout of children 0 Typically used for position but sometimes size 0 Dialog box OK Cancel buttons 39 stays at lower left l 15 Bottomup or insideout layout 0 Children determine layout of parent C Typically just size 0 Example Georgia 16 Bottomup or insideout layout 0 Children determine layout of parent 0 Typically just size 0 Shrinkwrap container 0 parent just big enough to hold all children Georgia Tech 0 eg pack method on JWindow and JFrame Resizes container to just big enough to accommodate contents preferredSizes 17 Q 555 Georgia 0 O O U 0000 Tech Q Q Which one is better 139quot 18 Neither one is suf cient 0 Need both 0 May even need both in same object 0 horizontal vs vertical C size vs position these interact 0 Need more general strategies Georgia 19 Layout Policies in Swing Swing layout policies are generally customizable Some containers come with a builtin layout policy 0 JSplitPaneJScrollPaneJTabbedPane Georgia Tech Others support pluggable policies through LayoutManagers Q LayoutManagers installed in Containers via setLayout 0 Two interfaces from AWT LayoutManager ancl LayoutManagerZ 0 Determines position and size of each component within a container 0 Looks at components inside container 0 Uses getMinimumSize getPreferredSize getMaximumSize 0 but is free to ignore these 0 Example LayoutManagers FlowLayout BorderLayout Grid Layout BoxLayout l 20 Georgia Tech Layout Policies in Swing 0 Each LayoutManager is free to do what it wants when layout out componens 0 Can ignore components minpreferredmax sizes 9 Can ignore not display components at all Generally most will look at children s requests and then 0 Size the parent component appropriately 0 Position the children within that component 0 Sotopdown with input from child components l 21 More general layout strategies 0 Boxes and glue model 0 Springs and struts model 0 Constraints Georgia Tech ll f l 22 Boxes and glue layout model Georgia Tech 0 Comes from the TeX document processing system I O Brought to UI work in Interviews toolkit C under X windows 0 See Composing User Interfaces with Interviews 0 Tiled composition no overlap 0 toolkit has other mechanisms for handling overlap 0 glue between components boxes 23 Georgia Tech Boxes and glue layout model 0 2 kinds of boxes hbox amp vbox 0 do horiz and vert layout separately 0 at separate levels of hierarchy 0 Each component has 0 natural size 0 min size 0 max size 24 Georgia Box sizes 0 Natural size Q the size the object would normally like to be 0 eg buttonztitle string border 0 Min size 0 minimum size that makes sense 0 egg button may be same as natural 0 Max size 25 Georgia Tech Boxes and glue layout model 0 Each piece of glue has 0 natural size min size always 0 max size often in nite stretchability factor 0 or in nite ok Stretchability factor controls how much this glue stretches compared with other glue l 26 Georgia Tech Example Paper pl 3 g 4amp5 0 Two level composition 0 vbox 0 middle glue twice as stretchable as top and bottom 0 hbox at top 0 right glue is in nitely stretchable 0 hbox at bottom 0 left is infinitely stretchable Georgia Tech How boxes and glue works 0 Boxes components try to stay at natural size 0 expand or shrink glue rst 0 if we can t t just changing glue only then expand or shrink boxes 0 Glue stretches shrinks in proportion to stetchability factor 28 Computing boxes and glue layout 0 Two passes 0 bottom up then top down 0 Bottom up pass 0 compute natural min and max sizes of parent from natural min and max of children 0 natural sum of children s natural 0 min sum of children s min 0 max sum of children s max Georgia 29 Georgia Tech Computing boxes and glue layout l 0 Top down pass 0 window size xed at top at each level in tree determine space overrun shortfall make up this overrun shortfall by shrinking stretching 0 glue shrunk stretched rst 0 if reaches min max only then shrink stretch components 30 Georgia Tech Top down pass cont 0 Glue is changed proportionally to stretchability factor 0 example 30 units to stretch 0 gluel has factor IOO 0 glue2 has factor 200 0 stretch gluel by IO 0 stretch glue2 by 20 0 Boxes changed evenly within min max 31 What if it doesn t t 0 Layout breaks 0 negative glue 0 leads to overlap Georgia Tech r 32 Georgia Tech Springs and struts model 0 Developed independently but can be seen a simpli cation of boxes and glue model 0 more intuitive has physical model 0 Has struts springs and boxes 0 struts are 0 stretchable glue 0 springs are infinitely stretchable glue 33 Georgia Tech Springs and struts model 0 Struts 0 specify a xed offset 0 Springs 0 specify area that is to take up slack 0 equal stretchability 0 Components boxes 0 not stretchable min natural max Georgia Tech Constraints 0 A more general approach 0 General mechanism for establishing and maintaining relationships between things 0 layout is one use 0 several other uses in Ul 0 deriving appearance from data 0 multiple view of same data 0 automated semantic feedback l 35 Georgia Tech General form declare relationships l 0 Declare what should hold 0 this should be centered in that 0 this should be l2 pixels to the right of that 0 parent should be 5 pixels larger than its children 0 System automatically maintains relationships under change 0 system provides the how 36 You say what Gegggci My System gures out how E 0 A very good deal 0 But sounds too good to be true L 4395 17 4 39aIquot 39 VIM A 39zp 1quot 0 You say what Georggg My System gures out how a 0 A very good deal 0 But sounds too good to be true 0 It is can t do this for arbitrary things unsolvable problem 0 Good news this can be done if you limit form of constraints 0 limits are reasonable 0 can be done very ef ciently 38 Georgia Tech Form of constraints 0 For Ul work typically express in form of equations 0 thisx thatX thatw 5 5 pixels to the right 0 thisx thatX thatw2 thisw2 centered o thisw IO max chidix chidiw l0 larger than children 39 Tech Georgia ME The Power of Constraints 0 thisx thatX thatw2 thisw2 0 What s so cool about this 0 Power comes from dynamic computation of result 0 Value isn t just computed immediater 0 Instead saves references to objects involved in calculation 0 When any operand changes result value is automatically recomputed 0 Express relationships declarativer 0 Systems updates as necessary to preserve the constraints you ve speci ed 40 Georgia Tech How would you express this thisx thatx thatw2 thisw2 0 Remember not programming language expression Parsable strings O c new Constraint thisx thatx thatw2 thisw2 Nested function calls 0 c new ConstraintEqualsthisxAddthisx SubDivthatw 2 Divthisw 2 Operator overloading 0 If your language supports it can make it look very like the example above 9 Requires de ning constraint objects overloading common arithmetic operators 41 Example doing springs and struts Geggg with constraints E Parent 42 Example doing springs and struts Geggsggig with constraints 3 Parent 0 First what does this do 0 Objl and obj3 stay xed distance from left and right edges 0 0ij centered between them 43 Example doing springs and struts Gegg j with constraints E Parent Compute how much space is left parentslack parentw obj w obj2w obj3w st w st2w 44 Example doing springs and struts Geggsg j with constraints E Parent 0 Space for each spring parentsplen parentsaclt 2 45 Example doing springs and struts Geggsggig with constraints 3 Parent 0 A little better version parentnumsp 2 if parentnumsp parentsplen 0 else parentsplen parentslack parentnumsp 46 Example doing springs and struts Geggg j with constraints E Parent 0 Now assign spring sizes sp w parentsplen sp2w parentsplen 47 Example doing springs and struts Geggg j with constraints E Parent 0 Now do positions left to right stX O objX stX stw spX objX objw 48 Power of constraints Georgia Tech 0 If size of some component changes system can determine new sizes for springs etc 0 automatically 0 just change the size that has to change the rest just happens 0 very nice property 49 Bigger example Georgia Tech l 0 Suppose we didn t want to x number of children etc in advance 0 don t want to write new constraints for every layout 0 instead put constraints in object classes has to be a more generaD 0 in terms of siblings amp rstlast child 50 Tech Georgia My Bigger generalized example 0 First compute slack across arbitrary children 0 Each strut spring and object 0 before means before considering this object 0 after means after considering this object 0 prevsibling is a name that dynamically refers to the object before obj at the same level in the tree if prevsibling null objslbefore parentw else objslbefore prevsiblingslafter 51 Tech Georgia My Bigger generalized example 0 For struts and objects 0 Roll forward subtracting out object sizes from slack objslafter objslbefore objw 0 For springs 0 Because they take up no space unless necessary springs don t detract from the slack sprslafter sprslbefore 52 Georgia Tech Example of a chained computation 0 Compute my value based on previous value 0 Special case at beginning 0 This now works for any number of children 0 adding a new child dynamically not a problem 0 Very common pattern l 53 Georgia My Tech Now compute number of springs 0 For springs use if prevsibling null sprnumsp l else sprnumsp prevsiblingnumsp l 0 For struts and objects use if prevsibling null objnumsp 0 else objnumsp prevsiblingnumsp 54 Georgia Tech Carry values to parent Propagate values computed in children up to the parent 39 lastchild is a dynamic reference that refers to the last child in the parent parentnumsp lastchildnumsp parentslack lastchildslafter 0 Again don t need to know how many children 0 Correct value always at last one 55 Georgia Tech Compute spring lengths 0 Figure up the length we ll use for each spring if parentnumsp parentsplen 0 else parentsplen parentslack parentnumsp 56 Set sizes of springs amp do positions 0 For springs use sprw parentsplen 0 For all use if prevsibling null objx 0 else objx prevsiblingx prevsibingw Georgia 57 Georgia Tech l More complex but 0 Only have to write it once Q put it in various superclasses this is basically all we have to do for springs and struts layout if we have constraints can also do boxes and glue slightly more complex but not unreasonable can write other kinds of layout and mix and match using constraints 58 Tech Georgia 4 Springs n Struts in Swing Swing provides a basic constraintbased Springs n struts LayoutManager 0 javaxswingSpringLayout 0 Allows simple arithmetic computation of constraints Georgia Tech Dependency graphs 0 Useful to look at a system of constraints as a dependency graph 0 graph showing what depends on what 0 two kinds of nodes bipartite graph 0 variables values to be constrained 0 constraints equations that relate Georgia Tech Dependency graphs 0 ExampleA fB C D A f 0 Edges are dependencies A r 5 s C Dependency graphs 0 Dependency graphs chain together g A f Y Georgia Tech X g AY U 62 Georgia Tech Kinds of constraint systems 0 Actually lots of kinds but 2 major varieties used in Ul work 0 reflect kinds of limitations imposed 0 OneWay constraints 0 must have a single variable on LHS 0 information only flows to that variable 0 can change BCD system will ndA 0 can t do reverse changeA OneWay constraints Results in a directed dependency graph 0 A fBCD B Alt f D Georgia Tech 0 Normally require dependency graph to be acyclic O cyclic graph means cyclic definition l 64 Georgia Tech OneWay constraints 0 Problem with oneway introduces an asymmetry thisX thatX thatw 5 0 can move change x that but not this i 65 Georgia Tech Multiway constraints 0 Don t require info flow only to the left in equation Q can changeA and have system nd BCD 0 Not as hard as it might seem 0 most systems require you to explicitly factor the equations for them 0 provide B gACD etc 66 Multiway constraints Georgia Tech 0 Modeled as an undirected dependency graph 0 No longer have asymmetry amp 67 Tech Georgia 4 Multiway constraints 0 But all is not rosy 0 most ef cient algorithms require that dependency graph be a tree acyclic undirected graph B x g A f OK 68 7 7 quot4 A l Tech 1 1439 w 1quot Georgia 4 Multiway constraints ButA fBCD amp X hDA B X h A f D Not OK because it has a cycle not a tree 69 Another important issue 0 A set of constraints can be 0 Overconstrained Georgia Tech 0 No valid solution that meets all constraints 0 Underconstrained 0 More than one solution 39 sometimes in nite numbers l 7O Georgia Tech Over and underconstrained 0 Overconstrained systems 0 solver will fail 0 isn t nice to do this in interactive systems 0 typically need to avoid this 0 need at least a fallback solution Over and underconstrained Tech Georgia My 0 Underconstrained many solutions system has to pick one may not be the one you expect example constraint point stays at midpoint of line segment 0 move end point then I 72 Over and underconstrained 0 Underconstrained 0 example constraint point stays at midpoint of line segment 0 move end point then 0 Lots of valid solutions 39 move other end point 39 collapse to one point 39 GtC Georgia Tech l 73 Georgia Tech Over and underconstrained 0 Good news is that oneway is never over or under constrained assuming acyclic 0 system makes no arbitrary choices 0 pretty easy to understand Over and underconstrained Georgia Tech 0 Multiway can be either over or underconstrained 0 have to pay for extra power somewhere i O typical approach is to overconstrain but have a mechanism for breaking loosening constraints in priority order 0 one way constraint hierarchies 75 Georgia Tech Over and underconstrained 0 Multiway can be either over or underconstrained 0 unfortunately system still has to make arbitrary choices 0 generally harder to understand and control Implementing constraints 0 Simple algorithm for oneway 0 Need bookkeeping for variables 0 For each keep 0 value the value of the var 0 eqn code to eval constraint 0 dep list of vars we depend on 0 done boolean mark for alg Georgia Tech 77 Simple algorithm for oneway 0 After any change reset all the marks for each variable V do Vdone false make each var up to date for each variable V do evaluateV Georgia 78 Georgia Tech Simple algorithm for oneway evaluateV if 1Vdone Vdone true Parms empty or each DepVar in Vdep do Parms evaluateDepVar Vvalue VeqnParms return Vvalue 79 Approach for multiway Gegrggg ig implementation 3 0 Use a planner algorithm to assign a direction to each undirected edge of dependency graph 0 Now have a oneway problem 80 Building Recognizers for Digital Ink and Gestures 0000006 000000 lb ooooudv comm Q Georgia Tech My E Digital Ink 0 Natural medium for penbased computing Q Pen inputs strokes G Strokes recorded as lists of XY coordinates 0 Egin Java 0 Point aStroke 0 Can be used as data handwritten content 0 or as commands gestures to be processed Georgia Tech H Distinguishing Content from Georgia My Tech Commands E 0 Depends on the set of input devices but 0 generally modal 0 Meaning that you re either in content mode or you re in command mode 0 Often a button or other model selector to indicate command mode 0 ExampleWacom tablet pen has a mode button on the barrel 0 Temporary switchonly changes mode while held down rather than a toggle Georgia Tech Other Options 0 Use a special character that disambiguates from content input and command input 3 Eg graf ti on PalmOS I U Command stroke says that what comes after is meant to be interpreted as a command a I 0 Can also have special alphabet of symbols that are unique to commands 0 Can also use another interactor eg the keyboard in but requires that you put down the pen to enter commands Georgia Tech Still More Options 0 Contextually aware commands 0 Interpretation of whether something is a command or not depends on where it is drawn 3 Eg lgarashi s Pegasus drawing beauti cation program 0 a scribble in free space is content 0 a scribble that multicrosses another line is interpreted as an erase gesture Georgia Tech Why Use Ink as Commands 0 Avoids having to use another interactor as the command interactor G Example don t want to have to put down the pen and pick up the keyboard 0 What s the challenge this with though 9 The command gestures have to be interpreted by the system i Needs to be reliable or undoablecorrectable 393 In contrast to content 0 For some applications uninterpreted content ink may be just ne Georgia Tech Content Recognizers 0 Featurebased recognizers 0 Canonical example Dean Rubine The Automatic Recognition of Gestures PhD dissertation CMU I990 3 Feature based recognizer computes range of metrics such as length distance between rst and last points cosine of initial angle etc Q Compute a feature vector that describes the stroke 0 Compare to feature vector derived from training data to determine match multidimensional distance function ii To work well requires that values of each feature should be normally distributed within a gesture and between gestures the values of each feature should vary greatly Georgia Tech H Content Recognizers 2 0 Unistrokes a la PalmOS Graf ti 0 Use a custom alphabet with highdisambiguation potential 0 Decompose entered strokes into constituent strokes and compare against template Q Eg unistrokes uses 5 different strokes written in four different orientations 045 90 and I35 degrees 0 Little customizability but good recognition ill E C l E l 39 339 quot 1 quot results and high data entry speed I If 3931P 1312 0 Canonical reference E quot UN DJ E39 if D Goldberg and C Richardson TouchTyping I L ET with a Stylus Proceedings of CHI I993 I Done I Georgia Tech U Content Recognizers 3 0 Waaaaay more complex types of recognizers that are out of the scope of this class 1 Eg neural netbased etc 10 quotiquot Georgia My Tech This Lecture 3 0 Focus on recognition techniques suitable for command gestures 0 While we can build these using the same techniques used for content ink we can also get away with some signi cantly easier methods Q Read hacks 0 Building generalpurpose recognizers suitable for large alphabets such as arbitrary text is outside the scope of this class 0 We ll look at two simple recognizers 9square 0 Siger 1O Georgia Tech 9square 0 Useful for recognizing Tivolilike commands 0 Developed at Xerox PARC for use on the Liveboard system Q Liveboard I992 4 foot X 3 foot display wall with pen input 0 Used in real life meetings over a period of several years supported digital ink and natural ink gestures 11 9 Square recognizer Georgia Tech 0 Basic version of algorithm Take any stroke 2 Compute its bounding box 3 Divide the bounding box into a 9square tictactoe grid 4 5 Compare this with a template Mark which squares the stroke passes through 12 I Original Stroke L 00 con 0000 Georgia 0 O fl 1 ch 23quot 13 2 Compute Bounding Box o o a one cocoa Georgla 0 O Tech f 1H 21 14 QOO m w 3 Divide Bounding Box into 9 eeggggiy Squares 3x3 grid 3 Q g3 F 15 4 Mark Squares Through Which Geggsggig the Stroke Passes E 5 1 2 3 4 5 6 representation X X X X 0 O X X X 7 8 9 uw G 30 m g g 16 5 Compare with Template stroke X X X X 0 O X X X template X X X X 0 O X X X Georgia Tech 17 Georgia Tech Implementing 9square 0 Create set of templates that represent the intersection squares for the gestures you want to recognize 0 Bound the gesture 9square it and create a vector of intersection squares 0 Compare the vector with each template vector to see if a match occurs 18 Georgia Tech Gotchas I 0 What about long narrow gestures like a vertical line 0 Unpredictable slicing Q A perfectly straight vertical line has a width of l impossible to subdivide Q More likely a narrow but slightly uneven line will cross into and out of the left and right columns 0 Solution pad the bounding box before subdividing Can just pad by a xed amount or Q Pad separately in each dimension 0 Long vertical shapes may need more padding in the horizontal dimension 0 Long horizontal shapes may need more padding in the vertical dimension 0 Compute a pad factor for each dimension based on the other 19 Gotchas 2 Hard to do some useful shapes eg vertical caret Is the correct template 0 x o 0 x o o X 0 or X o X X o X X 0 X or other similar templates lnherent ambiguity in matching the symbol as it is liker to be drawn to the 9square template Any good solutions Georgia Tech 20 Gotchas 2 Georgia Tech Hard to do some useful shapes eg vertical caret Is the correct template 0 X 0 0 X 0 o X 0 or X X X X 0 X X 0 X or other similar templates Inherent ambiguity in matching the symbol as it is liker to be drawn to the 9square template Any good solutions Represent that ambiguity Introduce a don t care symbol into the template 21 Georgia My Tech Don t Cares E 0 Use 0 to represent no intersection 0 Use X to represent intersection 0 Use to represent don t cares 0 Example 0 X 0 0 X 0 3 3 3 Or 3 X3 3 X 0 X X 0 X 0 Now need custom matching process simple equivalence testing is not smart enough 0 if strolltei templatei templatei 22 An Enhancement 0 What if we want direction to matter 0 Example M L Versus L Georgia Tech 23 Georgia Tech Directional NineSquares 0 Use an alternative stroketemplate representation that preserves ordering across the subsquares 0 Example r toptobottom 3 2 L4 7 8 9 1 2 3 0 bottomtotop 9 8 7 4 l 2 3 0 Can be extended to don t cares also 4 5 6 0 Treat don t cares as wild cards in the matching process 24 Sample 9square Gestures FJ AVltgtLW O with directional variants of each Georgia Tech Another Simple Recognizer 0 9square is great at recognizing a small set of regular gestures 0 but other potentially useful gestures are more dif cult 9 Example pigtail gesture common in proofreaders marks 0 Do we need to go to a more complicated real recognizer in order to process these 0 No 26 Georgia Tech The SiGeR Recognizer 0 SiGeR Simple Gesture Recognizer 0 Developed by Microsoft Research as a way for users to create custom gestures for Tablet PCs 0 Resources 9 httpmsdnmicrosoftcomlibrarydefaultaspurllibraryenus clntablethtmltbconCuGesRecasp U httpsourceforgenetprojectssiger C implementation 0 Big idea turn gesture recognition problem into a regular expression pattern matching problem 27 Georgia Tech Basic Algorithm Processes successive points in the stroke 2 Compute a direction for each stroke relative to the previous one and output a direction vector of the directions 3 Compare the direction vector to a pattern expression can even use standard regular expression matching 28 awe 00 I Process Successive Points in Geegaudy the Stroke E Waco 300 7 29 rwoooo r oo 2 Compute a direction vector Geegaudy based on each point quot U U U RU RU RU RU L L L LD D D RD RD RD D D D 3O 3 Compare the string to a Geggsg directionality template 3 U U U RU RU RU RU R R R RD D D LD LD LD D D D Template UPS RIGHTS DOWNS LEFTS DOWNS de nes basic shape of the stroke 31 De ning the Template Georgia Tech Concerned about matching 8 possible pen directions 1 RIGHT UP LEFT DOWN RIGHTUP RIGHTDOWN LEFTUP LEFT DOWN Template consists of these symbols plus grouping symbols that match more general directions 3 UPS matches all things that go up UP RIGHTUP LEFTUP 3 LEFTS matches all things that go left LEFT LEFTUP LEFTDOWN The template is then matched against the direction vector by seeing if the template patterns occur 32 Georgia Tech How Robust is This 0 Here s a gesture that shouldn t match but may depending on implementation 0 Why 3 A question mark appears in the middle of the stroke 0 Therefore 0 Important to match the whole stroke not just part of it 0 Think of the pattern as including A and regular expression markers for beginning of line and end of line at the rst and end 33 Georgia Tech How Robust is This 0 But requiring the entire stroke to match the pattern introduces a new problem 0 Can you tell what it is 34 Georgia Tech How Robust is This 2 0 But requiring the entire stroke to match the pattern introduces a new problem 0 Can you tell what it is 0 Look closer at the question mark 0 At the bottom the stroke jags off to the left 0 Common for the pen to make little tick marks like this when it comes into contact with the tablet or leaves it 35 Georgia Tech Solution 0 Simply trim the beginning and end points of the vector 0 More generally 9 Ignore small outlier points if the overall shape otherwise conforms to the shape pattern speci ed in the template 36 Implementing SiGeR one Gegg method 3 0 Specify some helper constants int UP I ltltO int DOWN lltltl de ne other constants as well as unique tokens that represent direction classes int RIGHTUP RIGHT UP int UPS UP RIGHTUP LEFTUP 0 Specify templates in code using humanreadable constants int QUESTIONMARK UPS RIGHTS DOWNS LEFTS DOWNS 37 Georgia Tech Implementing SiGeR continued 0 Create a function buildPatternString that takes the template and emits a regexp pattern that will be used to match it bufappend quot match the start of input bufappend 02 consume any character 02 times this gets rid of the noise at the beginning for int i0 iltpatternlength i switch patterni emit a unique letter code for each of the 8 directions case RIGHT bufappend R break case UP bufappend U break case RIGHTUP bufappend W break case LEFTUP bufappend X break case UPS bufappend UWX break combination directions combine letters bufappend 02 bufappend 38 Georgia Tech Implementing SiGeR Cont d 0 Write a function buildDirectionVector that takes an input stroke and returns a direction vector ill Compare each point to the point previous to it 0 Emit a symbol to represent whether the movement is UP RIGHT etc 9 using all of the 8 ordinal directions 0 Use the Java regular expression library to match strokes to patterns import javautilregex if questionMarkPatternmatcherstrokeString nd it s a question mark 39 PenBased Computing 000 00030 OODQ oooogg Georgia My Tech E Georgia Tech Agenda 0 Natural data types 0 PenAudioVideo 0 Penbased topics 0 Technology 0 Ink as data 0 Recognition Georgia Tech Natural Data Types 0 As we move off the desktop means of communication mimic natural human forms of communication 0 Writing nk 9 Speaking Audio Seeing Video 0 Each of these data types leads to new application types new interaction styles etc Pen Computing 0 Use of pens has been around a long time Georgia Tech l 6 Light pen was used by Sutherland before Engelbart introduced the mouse 0 Resurgence in 90 s 0 GoPad 0 Much maligned Newton 0 Types of pens 0 Passive same as using a nger Active pen provides some signal Example Pen Technology 0 Passive 0 Touchscreen eg PDA some tablets 0 Contact closure 0 Vision techniques 0 Active 0 Pen emits signals 0 eg IR ultrasonic 0 Where is sensing Surface or pen Georgia Tech l Questions about Pens 39 What operations detectable Contact updown DrawingWriting Hover Modi ers like mouse buttons Which pen used 0 Eraser 39 Difference between pen and mouse Georgia Tech i Example Expansys Chatpen 0 Reads clot pattern on paper 0 Transmits via Bluetooth Georgia Tech 0 httpwwwexpansyscomproductaspcodeERICCHATPEN Q Georgia Tech Example mimio 0 Active pens 0 IR ultrasonic 0 Portable sensor O Converts any surface to input surface 0 We have chained these mimic to create big surface 0 httpwwwmimiocom Georgia My Pen input Tech I Freeform ink mostly uninterpreted Tabet PC applications digital notebooks etc 2 Soft keyboards Provide highaccuracy although slow mechanism for inputting machineinterpretable text 3 Recognition systems Recognition of content Text handwriting recognition simpli ed textual alphabets Graphics doodles gures sketchbased interfaces Recognition of commands Specialized vocabulary of command symbols Modal input of commands Contextual commands commands distinguished from content only in how they are used Georgia Tech l ink as data when uninterpreted the easiest option to implement l Freeform ink ohumans can interpret otimestamping perhaps to support rollback undo oimplicit object detection gure out groupings crossings etc especialpurpose domain objects add a little bit of interpretation to some onscreen objects oEg Newton draw a horizontal line across the screen to start a new page oSee also Tivoli work Moran et al Xerox PARC 10 Freeform ink examples InkAudio integration Tivoi Xerox PARC eCass GT Fatand Xerox PARC Dynomite FXPAL The Audio Notebook MIT Georgia Georgia Tech U 00 2 Soft Keyboards Make recognition problem easier by forcing users to hit specialized onscreen targets Sometimes a blurry line between what s recognition and what s a soft keyboard common on small mobile devices many varieties otapping interfaces oKey layout QWERTY alphabetical learnability vs ef ciency 12 Georgia Tech T9 Tegic Communications oAlternative tapping interface oPhone layout plus dictionary oSoft keyboard or mobile phone oNot usually pen based but ideas for rapid text entry often carry over from fingertips to pens 13 Tech Georgia 4 Quickwrite Perlin Unistroke recognizer 0 Start in rest zone center 0 Each character has a major zone large white areas 0 and a minor zone its position within that area 0 To enter characters in the center of a major zone move from the rest zone to the character s major zonethen back 0 Example for A move from rest to upper left zone then back to rest 0 To enter characters at other points in a zone move into the character s major zone then into another major zone that corresponds to the character s minor zone 0 Example F is in the topright zone its major zone Move from rest to that major zone Since F is in the topcenter of its major zone move next into the topcenter major zone then back to rest 0 Allows quick continual writing without ever clicking a mouse button or lifting the stylus 14 000 0000 00 Georgia 000 Tech 4 CIrrIn Mankoff amp Abowd Wordlevel unistroke recognizer Ordering of characters minimizes median distance the pen travels d based on common letter pairings quot3 I k b r r I q 9 I nished x hi i p t 39 j j a v 3 39 I z m i c 15 Georgia Tech l Unlike soft keyboards recognize more natural pen strokes 3 Recognizing pen input Can be used for both content and commands Some are less natural than others Graf ti unistroke alphabet Other pen gesture recognizers for commands oStanford flow menus PARC Tivoli implicit objects omeasure features of strokes Rubine Long ousually no good for complex strokes 16 Handwriting content recognition Lots of resources see Web good commercial systems Two major techniques onine as you write ooffIine batch mode Which is harder Georgia 17 Handwriting content recognition Lots of resources see Web good commercial systems Two major techniques online as you write ooffline batch mode Which is harder Georgia Tech H Offline You don t have the realtime stroke information direction ordering etc to take advantage of in your recognizer only the nal ink strokes 18 Georgia Tech l Mixing modes of pen use Users want freeform content and commands oor commands vs text How to switch between them I mode recognize which applies contextual commands a la Tivoli Teddy etc 2 modes visible mode switch Graf ti make special command gesture l 5 modes special pen action switches temporary or transient mode egWacom tablet pens 19 Georgia Tech Error correction Necessary when relying on recognizers which may often produce incorrect results Ul implications even small error rates l can mean lots of corrections must provide Ul techniques for dealing with errors Really slows effective input wordprediction can prevent errors Various strategies repetition erase and write again nbest list depends on getting this from the recognizer as confidence scores other multiple alternative displays 20 Other interesting applications Signature veri cation Notetaking group NotePals by Landay Berkeley student StuPad by Truong GT meetings Tivoli and other commercial Sketching systems early storyboard support SILK Cocktail Napkin sketch recognition Eric Saund PARC others Georgia Tech l 21 Toolkits for PenBased Interfaces Georgia Tech SATIN Landay and Hong Java toolkit MS Windows for Pen Computing MS Pocket PC CEnet Apple Newton OS GO PenPoint Palm Developer environments GDT Long Berkeley Javabased trainable unistroke gesture recognizer OOPS Mankoff GT error correction 22 SATIN UIST 2000 0 Pen input for informal input 0 Sketching others have investigated this Common toolkit story 0 Gee X sure is a neat class of apps 0 Golly making X apps is tough 0 Here s a toolkit to build X things easily Georgia Tech l 23 Sound and NonSpeech Interfaces Going Beyond Conventional GUls oooooi oooo o Georgia Tech My E Audio Basics Georgia Tech amp oo oo 00 Georgia on Tech 39 How sound IS created Sound is created when air is disturbed usually by vibrating objects causing ripples of vibration varying air pressure propagated by the collision of air molecules l QRJ Pressure waves in air Rafi de 2fr Georgia Tech Why Use Audio Good support for offthedesktop interaction 9 Handsfree potentially 0 Display not necessary 0 Effective at a short distance 0 Can add another information channel over visual presentation l How Sound is Perceived Georgia Tech 0 Characteristics of physical phenomenon the sound wave How we perceive those Amplitude Frequency Volume Pitch l 47 1 Second 1 Cycle 1 Cycle 2 Cycle 3 Cycle 4 M 1 srllfglitlltle a Po AElm Displacem em v r i 13p acemam T I Georgia Tech Complex Sounds Most natural sounds are more complex than simple sine waves 0 Can be modeled as sums of more simple waveforms or put another way G More simple waveforms mix together to form complex sounds utterance of quotDee ES Peequot Wave 1 WEME 2 t time in sec co co o Georgla on Tech 0 E 0 Sampling AUClIO 0 Sampling rate affects accurate representation of Xe sound wave 3quot Nyquist sampling theorem r if mire if Must sample at 2x the Elf quotc maximum possible frequency I to accurater record it i ilj H time 0 Eg 44OO Ha sampling If rate CD quality can V capture frequencies up to 22050 HZ Additional Properties ofAudio that can be Exploited to Good Ge i ll Effect E Sound localization 0 Auditory illusions Tech Georgia My Sound Localization 0 We perceive the location of where a sound originates from by using a number ofcues Q Interaural time delay the difference between when the sound strikes left versus right ears Q Perhaps most important headrelated transfer function how the sound is modi ed as it enters our ear canals 0 We can take a normal sound and process it to recreate these effects D Calculate and add precise delay between left and right channels 8 Apply a lter in realtime to simulate HRTF i Requires ability to pipe different channels to left and right ears 0 Problematic each person s HRTF is slightly different Q Because of external ear shape 0 Still can do a reasonably good job 0 Generally need head tracking to keep apparent position fixed as head moves Georgia Tech Auditory Illusions 0 Example Shepard Tone 0 Sound that appears to move continuously up or down in pitchyet which ultimately grows no higher or lower 9 Identi ed by Roger Shepard at Bell Labs l960 s Useful for feedback where you have no bounded valuator Georgia Tech Speech versus nonspeech audio Speech is just audio why consider them separately 0 Uses in interfaces are actually vastly different more on this later O Actually processed by different parts of the brain 0 Understanding the physical properties of audio you can create new interaction techniques 0 Example cocktail party effect being able to selectively attend to one speaker in a crowded room 0 Requires good localization in order to work 0 In this lecture we re focusing largely on nonspeech audio Georgia Tech Using Audio in Interfaces 0 That s all ne but what special opportunitieschallenges does audio present in an interface 12 Georgia Tech Changing the assumptions What happens when we step outside the conventional GUI desktop widgets framework 0 Topic of lots of current research 390 Lots of open issues But a lot of what we have seen is implicitly tied to GUI concepts 13 Georgia Tech Example I nte ractive TV 0 WebTV and friends 0 Idea is now mostly dead but was attempt to add a return channel on cable and allow the user to provide some input 0 Basic interactionthough is similar for Tivo ancl other living room interfaces 0 Is this just another GUI Why or why not 14 Not just another GUI because 0 Why Georgia Tech Q QC 0 O 15 Georgia My Tech Not just another GUI because 0 Remote control is the input device 6 Not a decent pointing device 0 Despite having many dimensions of inputpotentially one for each button 0 Context amp content is different 0 Couch potato mode 0 only a few alternatives at a time 0 simple actions 0 the ten foot interface no ne detail not that you have the resolution anyway 39gt Convenient to move in big chunks 16 PreVI ew Geegall Leads to a navigational approach E Have a current object Act only at current object 0 Typically small number of things that can be clone at the object 0 Often just one Move between current objects 17 ExamplezTivo 0 UPDOWN 0 Moves between programs 0 LEFTRIGHT 0 Moves to menussubmenus At each item there are a small xed set of things you can do 0 SELECT it 0 DELETE it 0 maybe a few others depending on context Tech Georgia 4 Now Playing List L Footballers Wwes Ovettime Sun Q Piay It as It Lays Sun 86 0 Saturday Nign Live Sat 85 A Flea in He Ear Sat 85 a 0 Late Night With Conan O39Bn en Sat 85 lt1 1110 Sou F 34 V I 0 Kath amp Kim F 84 sundancc I The Wand Stands Up 2 M 84 wwnuw bw 39 18 Generalizing Nonpointing input Georgia Tech l 0 In general a lot of techniques from GUls rely on pointing 3 Example a lot of input delivery 0 What happens when we don t have a pointing device or we don t have anything to point to 0 Extreme exampleAudio only 19 o o o The Mercator System Georgia 0 w httpwwwacmorgpubscitationsproceedingsuist TeCh E g 4262 lp6 I mynatt 0 Designed to support blind users of GUls 0 GUls have been big advance for most 0 Disaster for blind users 0 Same techniques useful for eg cell phone access to desktop 0 Converting GUI to audio 20 ChallengezTranslate from visual into audio 0 Overall a very dif cult task 0 Need translation on both input and output Georgia Tech 1quot 0 I ll 4 J 39iquot lfill 21 Output translation Georgia Tech l 0 Need to portray information in audio instead of graphics hard Not a persistent medium 0 Much higher memory load Sequential medium 0 Can t randomly access Not as rich high bandwidth as visual 0 Can only portray 23 things at once 0 One at a time much better 22 Georgia Tech Mercator solution 0 Go to navigational strategy 3 only at one place at a time 0 only portray one thing at a time 0 But how to portray things 0 Extract and speak any text Q Audio icons to represent object types Audio icons 0 Sound that identi es object Georgia Tech 6 eg buttons have characteristic identifying sound 0 Modi ed to portray additional information 0 Filtears manipulate the base sound i 24 Filtear examples 0 Animation 0 Accentuate frequency variations 0 Makes sound livelier 0 Used for selected 0 Muffled 0 Low pass lter 0 Produces duller sound 0 Used for disabled Georgia Tech l 25 Georgia Tech l Filtear examples 0 Inflection 0 Raise pitch at end 0 Suggests more like questions in English 0 Used for has submenus 0 Frequency 9 map relative location eg in menu to change in pitch high at top etc 26 Georgia Tech Filtear examples 0 Frequency Reverberation 0 Map size eg of container to pitch big low and reverb big lots 0 These are all applied over the top of the base audio icon 0 Can t apply many at same time Tech Georgia Mapping visual output to audio 3 0 Audio icon design is not easy 0 But once designed translation from graphical is relatively straight forward 0 eg at button play button icon speak textual label 0 Mercator uses rules to control 0 when you see this do that 28 Also need to translate input 0 Not explicit but input domain also limited 0 Nothing to point at can t see it O Pointing device makes no sense 0 Again pushes towards navigation approach 0 limited actions move act on current 0 easily mapped to buttons Georgia 29 Georgia Navigation 0 What are we navigating 6 Don t want to navigate the screen 0 very hard useless wo seeing it 0 Navigate the conceptual structure of the interface 0 How is it structured at Ul level 0 What it is at interactor level 30 Georgia Tech Navigation 0 But don t have a representation of the conceptual structure to navigate 0 Closest thing interactor tree 0 Needs a little tweaking Navigate transformed version of interactor tree Georgia Tech Transformed tree 0 Remove purely visual elements 6 separators and decoration 0 Compress some aggregates into one object 6 eg message box with OK button 0 Expand some objects into parts 0 eg menu into individual items that can be traversed l 32 Georgia Tech Traversing transformed tree 0 Don t need to actually build transformed tree 0 Keep cursor in real interactor tree 0 Translate items skip etc onthefly during traversal 0 Traversal controlled with keys 0 up rstchild nextsibling prevsibling top l 33 Traversing transformed tree Georgia Tech l 0 Current object tells what output to create amp where to send input 0 upon arrival play audio icon text 0 can do special purpose rules 0 Have key for do action 0 action speci c to kind of interactor G for scrollbar only need two keys 34 Other interface details 0 Also have keys for things like 3 repeat current O play the path from the root 0 Special mechanisms for handling dialog box 0 have to go to another point in tree and return 0 provide special feedback Georgia Tech l 35 Mercator actually has to work a Geere ll bit harder than I have described E 0 Xwindows toolkits don t give access to the interactor tree 0 Only have a few query functions listening to the wire protocol 0 protocol is low level 0 drawing events window actions 36 a Georgia Mercator actually has to work Tech bit harder than I have described 0 Interpose between client and server 0 query functions get most of structure of interactor tree 0 reconstruct details from drawing commands 0 catch amp modify events l r 37 Georgia Tech Why not just use a toolkit that gives access to the tree l 0 To be really useful Mercator needed to work on existing off the shelf applications without modi cation not even recompile or relink 0 critical property for blind users 38 Tech Georgia My Audio Input Most audio input has focused on speech input However some work on nonspeech input 0 Example O Voice as Sound Using NonverbaIVoice Input for Interactive Control Igarashi and Hughes UIST 200I 39 Question for the class 0 How would you do drag and drop in audio Georgia 4 39 L 21 4 i n 1 71 4O I O o Georgla 00 Tech 039 R E JavaSound architecture File b I J H Speaker 39 39 Natwurkl 0mm Line lit 6 Insanely good and underutilized iii I H Device Hmjlm architecture for doing powerful 39 APPhmtion H h In 4 Micmphone audio processmg Film ngce PM Network 41 4 l Inputcmmel Input o 1 BUIlCl on technology licensed from Beatnik lnc Mixer Output w 13quot123322 9 Company started byThomas Dolby the blinded me with science guy httpjavasuncomproductsjavamediasound httpwwwjsresourcesorg Input part 3 Implementing Interaction Techniques ii COED COOQ OOOOQ O 0 Georgia Tech Georgia Tech Recap Interaction techniques 0 A method for carrying out a speci c interactive task 0 Example enter a number in a range 0 could use simulated slider 0 simulated knob 0 type in a number text edit box 0 Each is a different interaction technique l Suppose we wanted to Georgia on Implement an Interaction for Tech g specifying a line 0 Could just specify two endpoints 39 click click 0 not good no affordanceno feedback 0 Better feedback is to use rubber banding 0 stretch out the line as you drag O at all times shows where you would end up if you let go Aside 0 Rubber banding provides good feedback 0 How would we provide better affordance Georgia Tech Q Aside 0 Rubber banding provides good feedback 0 How would we provide better affordance 0 Changing cursor shape is about all we have to work with Georgia Tech i Georgia 1bch Implementing rubber banding Accept the press for endpoint pl P2 Pl Draw line Pl PZ Repeat Erase line Pl PZ P2 currentposition Draw line Pl PZ Until release event Act on line input l Tech Georgia 4 Implementing rubber banding 0 Need to get around this loop absolute min of 5 times sec 0 l0 times better 0 more would be better 0 Notice we need undraw here Georgia Tech What s wrong with this code Accept the press for endpoint pl P2 P1 Draw line Pl P2 Repeat Erase line Pl P2 P2 currentposition Draw line Pl P2 Until release event Act on line input l Tech Georgia My Not event driven 0 Not in the basic event redraw cycle form Q don t want to mix event and sampled in many systems can t ignore events for arbitrary lengths of time 0 How do we do this in a normal event redraw loop You don t get to erte control lt eltsglenilj flow anymore 3 0 Basically have to chop up the actions in the code above and redistribute them in event driven form C event driven control flow 0 need to maintain state where you are between events and start up in the state you were in when you left off 10 Georgia Tech i 0 One good way to maintain state is to use a state machine Finite state machine controllers 0 deterministic finite state machine 0 FSM Tech Georgia My FSM notation E 0 Circles represent states O arrow for start state 0 double circles for nal states 0 notion of nal state is a little off for user interfaces don t ever end 0 but still use this for completed actions 0 generally reset to the start state 12 Georgia Tech FSM notation O Transitions represented as arcs O Labeled with a symbol 0 for us an event can vary 0 Also optionally labeled with an action MouseDn DrawLine 13 Tech Georgia My FSM Notation MouseDn DrawLine 0 Meanszwhen you are in stateA and you see a mouse down do the action call drawine and go to state B 14 FSM Notation 0 Sometimes also put actions on states 0 same as action on all incoming transitions Georgia Tech l 15 Rubber banding again a g cutting up the code 2 Accept the Eress for endpoint p1 A P2 P1 Draw line Pl PZ Ke eat B Erase line P1 P2 P2 currentposition Draw line Pl PZ Untll release event CzlAct on line input 16 FSM control for rubber banding Ge g g Move B Press A A P2 P1 raw line Pl PZ B Erase line Pl PZ P Release C 2 currentposition raw line Pl PZ C IAct on line inputl 17 Georgia Tech Second example button highlight change highlight Press inside Move inout Release inside act llll Release outside do nothing 18 Georgia Tech My FSM for a button FSM for a button Enter C Georgia Tech Q Release D Leave B Pressinside A Release E Georgia Tech ReleaseD FSM for a button Leave B Enter C Pressinside A Release E A highlight button B unhighlight button C highlight button D ltdo nothinggt E do button action 21 In general Georgia Tech 0 Machine states represent context of interaction 0 where you are in control flow 0 Transitions indicate how to respond to various events 0 what to do in each context 22 Georgia Tech Events in FSMs 0 What constitutes an event varies Q Q may be just low level events or higher level synthesized events 0 eg regionenter pressinside Example Swing Action Events 0 Generated from a range of different lowlevel events Completion of button activation FSM Hitting enter in a text field l 23 Georgia Tech Guards on transitions 0 Sometimes also use guards Q predicate boolean expression before event 0 adds extra conditions req to re Q typical notation pred event action 0 eg buttonenabed pressinside A l 0 Note FSM augmented with guards is Turing complete 24 FSM are a good way to do control flow in event driven Geggggil 39 svstems 0 Can do formal or informal analysis 0 are all possible inputs eg errors handled from each state 0 what are next legal inputs 0 can use to enable disable 0 Can be automated based on higher level speci cation 25 0 O 0 Georgia 000 Tech 39 Implementing FSMs state startstate for rawevt waitforevent evt transformeventrawevt state fsmtransitionstate evt 0 Note that this is basically the normal event loop 26 CO Georgia My 000 Tech 39 Implementing FSMs Q fsmtransitionstate evt switch state case 0 case for each state case 1 case for next state return state 27 Implementing FSMs switch evtkind action m case locdn switch evtkind case locmove state 42 Georgia Tech trans evt trans action trans target 28 0 O 0 Georgia 000 Techig 0 Implementing FSMs E fsmtransitionstate evt switch state case 0 case for each state switch evtkind case locmove trans evt action m trans action state 42 trans target case locdn case 1 case for next state switch evtkind return state 29 Georgia Tech l Table driven implementation 0 Very stylized code 0 Can be replaced with xed code table that represents FSM Q only have to write the xed code once 0 can have a tool that generates table from something else 30 Table driven implementation 0 Table consists of array of states 0 Each state has list of transitions 0 Each transition has 0 event match method 0 list of actions or action method G target state Georgia 31 Architecture and Organization of UI Software 3 Ways To look at it Layers Detailed tasks I What the code looks like View 1 Break UI Tasks into layers Whole Application Hardware w I Output raster display device I Input keyboard amp mouse others I pointing devices trackball etc I knobs and sliders banks of buttons I touch screens tablets eg pen input 3D locators Voice input DataGlove Eye tracking 05 Window system Toolkits I OS Not really concerned about it here I Window system provides a virtual device interface I Toolkits I Library of reusable interface components Central level we program in Higher level tools r 7 7 I Automate aspects of producing UIs Different names User Interface Management Systems UIMS User interface builders User interface designdevelopment environments Whole Application I Often distinguish between I UI parts quotcore functionality application I Carry out actions on behalf of user I UI has two interfaces one w user one w application I Challenge balancing demands of each Strong or weak separation A r 7 iii I SE view a strong separation is good I UI view may not be I user doesn t knowcare UI s take large of code I hard to tell bits apart I Separation of concerns View 2 Detailed tasks I Task oriented vs systems oriented I What does a U1 need to do How do we organize SW to do it I Partially depends on UI stylemetaphor I Two dominant metaphors for U1 I conversationa or anguage metaphor l the smuated ward metaphor The conversational metaphor r 7 7 iii I Aim analogous to human conversation User quottalkingquot to computer in a language Computer responding in some language Le command and menu style interfaces I Dominant approach till early 90 s Interface as a Language Interpreter I Lexical accept low level inputs I Syntactic parse to assign structure I Semantic assign meaning act on them I Produce results in quotoutput languagequot I Inputoutput media are typically related I quotInterreferential IOquot An architectural model based on the language metaphor Ni 31 w H The Made uquot I Presentation lexical level 10 I Dialog syntactic level dialog mgmt Input in context where why next I Application interface input translation output abstraction Other major metaphor Simulated Worlds I UI as collection of simulated objects I Direct manip of objects vs conversation I Avoid conversational intermediary I Conversation hasn39t really gone away I illusion of directness I same kind of tasks on a per object basis Seeheim architectural model revisited H I Same parts but dialog less imp I per object basis vs whole system I syntax localized hence simple I Separation of Concerns again I SE we want to separate concerns I DM metaphor want to mix into each object I return to this again in toolkits OO methods Underlying Commonality I The same central task I UI is a mechanism that translates manipulation of input devices into actions carried out for the user View 3 What does the code look like Application Level User ltgt UI ltgt Appl Information flow user gt input appl output L User Side Input events I Output photons Application Side H I Input and Output procedure or method calls I Input side callbacks I Recall conversational metaphor input gt commandsactions I UI code is similar I input gt procedure calls or messages Callbacks imply a major restructuring of app code I quotNormalquot code provides all control flow UI code is acting on behalf of the user gt main control flow in user hands I Code is reactive rather than proactive l quotevent driven programmingquot I Usually harder I Programmer has little control harder to modularizestructure So what does This look like in the code I Initialize I Repeat I Wait for next user action expressed as quotevent recordquot Decide what it means I Act on it Update the screen to reflect the changes I Until done Building Recognizers for Digital Ink and Gestures 0000006 000000 lb ooooudv comm Q Georgia Tech My E Digital Ink 0 Natural medium for penbased computing Q Pen inputs strokes G Strokes recorded as lists of XY coordinates 0 Egin Java 0 Point aStroke 0 Can be used as data handwritten content 0 or as commands gestures to be processed Georgia Tech H Distinguishing Content from Gegggg Commands E 0 Depends on the set of input devices but 0 generally modal 0 Meaning that you re either in content mode or you re in command mode 0 Often a button or other model selector to indicate command mode 0 ExampleWacom tablet pen has a mode button on the barrel 0 Temporary switchonly changes mode while held down rather than a toggle Georgia Tech Other Options 0 Use a special character that disambiguates from content input and command input 1 393 Eg graf ti on PalmOS 0 Command stroke says that what comes after is meant to be interpreted as a command 6 a 9 O 0 0 Can also have special alphabet of symbols that are unique to commands 0 Can also use another interactor eg the keyboard in but requires that you put down the pen to enter commands Georgia Tech Still More Options 0 Contextually aware commands 0 Interpretation of whether something is a command or not depends on where it is drawn 3 Eg lgarashi s Pegasus drawing beauti cation program 0 a scribble in free space is content 0 a scribble that multicrosses another line is interpreted as an erase gesture Georgia Tech Sketchbased user interfaces 0 User interfaces aimed at creating L f re ning and reusing handdrawn f if 39 llhl A 39 k i 1 Input 1335 213 kammg 39 391 Lee 1 af rms 31 39 a 39 o p 1 quot a3939lu l3h j39 l 39 39 axe lb riquot Typlcally what Wu erzw E1771 0 Few normal GUI controls 3ng i if gigt C E L 392 ll CTIF PJri 3 Strokes contextually Interpreted and quotW3 39 varm lJ lw l mtAl Intermlngled WIth content L quotligttz 39 0 Examples i Drawing beauti cation lgarashi Pegasus 0 UI creation Landay SILK 0 Turn UML diagrams etc into machine representations Saund 0 3D modeling lgarashizTeddy Georgia Tech Why Use Ink as Commands 0 Avoids having to use another interactor as the command interactor G Example don t want to have to put down the pen and pick up the keyboard 0 What s the challenge this with though 9 The command gestures have to be interpreted by the system i Needs to be reliable or undoablecorrectable 393 In contrast to content 0 For some applications uninterpreted content ink may be just ne Georgia Tech Content Recognizers 0 Featurebased recognizers 0 Canonical example Dean Rubine The Automatic Recognition of Gestures PhD dissertation CMU I990 3 Feature based recognizer computes range of metrics such as length distance between rst and last points cosine of initial angle etc Q Compute a feature vector that describes the stroke 0 Compare to feature vector derived from training data to determine match multidimensional distance function ii To work well requires that values of each feature should be normally distributed within a gesture and between gestures the values of each feature should vary greatly Georgia Tech H Content Recognizers 2 0 Unistrokes a la PalmOS Graf ti 0 Use a custom alphabet with highdisambiguation potential 0 Decompose entered strokes into constituent strokes and compare against template 9 Eg unistrokes uses 5 different strokes written in four different orientations 045 90 and I35 degrees 0 Little customizability but good recognition E f D F l l results an d high data entry speed I zz2 LEV lfxi 1 P j 112 0 Canonical reference 33 1 L 1 ti 12 3 D Goldberg and C Richardson TouchTyping m I 3 3 L c l3 1 1 with a Stylus Proceedings of CHI I993 Ifquot 5 539 Done Georgia Tech U Content Recognizers 3 0 Waaaaay more complex types of recognizers that are out of the scope of this class 1 Eg neural netbased etc WC CCCCCC ICCCCC Georgia My Tech This Lecture 3 0 Focus on recognition techniques suitable for command gestures 0 While we can build these using the same techniques used for content ink we can also get away with some signi cantly easier methods Q Read hacks 0 Building generalpurpose recognizers suitable for large alphabets such as arbitrary text is outside the scope of this class 0 We ll look at two simple recognizers 9square 0 Siger 11 Georgia Tech 9square 0 Useful for recognizing Tivolilike commands 0 Developed at Xerox PARC for use on the Liveboard system Q Liveboard I992 4 foot X 3 foot display wall with pen input 0 Used in real life meetings over a period of several years supported digital ink and natural ink gestures 12 9 Square recognizer Georgia Tech 0 Basic version of algorithm Take any stroke 2 Compute its bounding box 3 Divide the bounding box into a 9square tictactoe grid 4 5 Compare this with a template Mark which squares the stroke passes through 13 I Original Stroke L 00 con 0000 Georgia 0 O fl 1 ch 23quot 14 2 Compute Bounding Box o o a one cocoa Georgla 0 O Tech f 1H 21 15 QOO m w 3 Divide Bounding Box into 9 eeggggiy Squares 3x3 grid 3 Q g3 F 16 4 Mark Squares Through Which Geggsggig the Stroke Passes E 5 1 2 3 4 5 6 representation X X X X 0 O X X X 7 8 9 uw G 30 m g g 17
Are you sure you want to buy this material for
You're already Subscribed!
Looks like you've already subscribed to StudySoup, you won't need to purchase another subscription to get this material. To access this material simply click 'View Full Document'