LMS Focus Group Report

LMS Focus Group Report

• 1.
  LMS
Focus
Group
Report
 
 Background
 
 UT Dallas’ currenteLearning system, Blackboard Vista 8.2, is an end-of-life system and will no longer be supported by Blackboard after 2013. With this end in mind, the university is conducting a search for a new learning management system. UTD will publish a Request for Proposals in early Spring 2011. In October and November 2010, as directed by the LMS Committee, the Office of Educational Enhancement, assisted by SOM eLearning, conducted a series of focus groups to solicit feedback from faculty, staff and students about the use of the current eLearning system and about their needs to determine what features would be necessary and desired in any learning management system. 
 Focus
Groups
 A total of eight (8) focus groups were held - 3 for faculty users, 3 for student users, 1 on accessibility and 1 for back-end users. See the below tables for attendance and demographic information of attendees. At the beginning of each focus group (except for the back-end users group), a brief survey was administered individually to solicit quick feedback about specific tools used. A comments area was provided. See Appendix A for the survey and survey results. Guiding
Questions
 For the focus group sessions, a set of “guiding questions” was used to direct the conversations. This semi-structured approach allowed the facilitator to guide the discussion and to prevent the focus groups from becoming “gripe” sessions. See Appendix B for the Guiding Questions. Observations
 Interestingly, many of the features that faculty expressed a desire for in a new system already exist in the current eLearning. Faculty were either unaware of them or had not received training. Faculty emphasized that migration from eLearning to new system should be easy. Students generally wished that all faculty used or were required to use eLearning. “It would be convenient.” Not all of the students were taking full online course; thus their faculty tended to use eLearning for posting syllabi and lecture notes and homework assignments. Some students took online quizzes. Student disappointment with eLearning was more from lack of use by faculty and from stability. Students related that other students don’t like eLearning because it often didn’t work and they need to feel confident that the system will work.
• 2.
  
 Focus
Group
Results
 
 Current
usage
(from
surveys)
 See
Appendix
A
 
 Usability
 • “eLearning
is
powerful
but
for
a
power
user”
(F)
 • “the
learning
curve
is
more
work
than
it’s
worth”
(F)
 •navigation
is
easy
(S)
 • could
look
more
modern,
more
updated
(S)
 
 Tools

 
 Communication
 Mail
 • Tool
should
be
a
default
in
all
courses
(F)
 • Need
search
by
name
and
keyword
(F)
 • If
mail
forwarded
out,
need
to
be
able
to
reply
back
into
(F)
 • Window
doesn’t
expand
(F)
 • Connect
UTD
email
and
eLearning
mail
(F)
(S)
 Discussions

 • Not
intuitive
to
create
a
new
post

(F)
 • In
graded
discussions,
move
from
student
to
student
(not
back
to
 list)
(F)
 • Need
to
reply
privately
to
a
post
(F)
 • grading
is
clunky
(F)
 • doesn’t
that
it
opens
new
into
separate,
smaller
windows
(F)
 • needs
to
be
easy
to
use
(like
Facebook
messaging)
(S)
 • in
“show
all”
should
still
be
threaded
(S)
 • “read”
should
apply
only
to
ones
actually
read
(not
all
new)
(S)
 • searchable
and
tagable
threads
(S)
 Calendar
 • link
to
Outlook
calendar
/
Google
calendar
(F)
 • color
code
entries
by
course
(user
assigned
colors)
(F)
 Announcements
 • need
ability
to
edit,
delete
announcements
by
TAs
(F)
 • need
ability
to
hide
announcements
(F)
 Chat
 • would
be
essential
to
online
course
(S)
 • chat
not
sufficient
(moved
to
Skype)
(S)
 
 
 

• 3.
  Collaboration
 Groups
 • useful
(F)
 • manual
creation
of
groups
is
somewhat
hard
(F)
 • nice
but
not
crucial
(S)
 • group
contact
could
be
expanded
(S)
 
 Assessment
 Assignments
 • Easy
formatting
w/o
launching
html
editor
(F)
 • Sequential
grading
of
assignments
-
student
to
student
(no
return
 to
list)
(F)
 • Easy
upload
of
multiple
or
large
files
(eg.
video)
(F)
 • Somewhat
difficult
to
submit
(S)
 • Some
files
(MS
Access)
hard
to
download
(S)
 Quizzes
 • Improve
graphic
ability
(embedding,
formats)
(F)
 • Change
settings
while
quiz
is
open
(F)
 • Equation
editor
isn’t
good;
integrate
“MathType”
(F)
 • System
not
“dynamic”,
though
works
well
for
MC
tests
(S)
 Grading
forms
 • Ability
to
add
comments
with
each
item
(F)
 
 Learning
Modules
 
 • More
visual
way
(F)
 
 Course
Management
 
 • Desired
-
central
location
from
which
to
push
email,
announcements,
 exams
to
all
sections
(F)
 File
manager
 • Should
work

more
like
a
drop
box
(F)
 
 • Faster
download/uploading
of
files
(F)
 Syllabus

 • Integrate
with
Coursebook
(F)
 • Hypertext
to
calendar
(F)
 Icons
 • More
choices
(F)
 Edit
pages
 • Make
default
“See
all”
(F)
 MyFiles
 • Nice,
files
always
there
(S)
 
 Gradebook
 Columns
 • Rearranging
is
difficult
(F)
 • Need
per
column
settings
(F)
 Formulas
 • Formula
editing
is
difficult
(F)
 General
 • Add
categories
(F)
 • Have
default
grading
schemes
(F)
 • Make
it
more
like
Excel
(F)

• 4.
  
 Integrated
Tools
 • Library,
library
reserves
(F)
(S)
 • Video
server,
YouTube,
iTunesU,
etc
(F)
 •Google
Docs
(F)
 • Turnitin
-
use
Grademark
(online
grading)
(F)
 • MuchLearning
(calculus)
(S)
 • Skype
(S)
 
 Additional
tools
or
functionality

 • Online
editing
of
documents,
regardless
of
format
(F)
 • Better
text
editor
(F)
 • Notification
system
(send
reminders
after
missed
exam,
send
new
announcement,
 new
discussion,
etc,
RSS
feeds)
(F)
 • Push
notification
for
updates
(S)
 • Ability
to
embed
videos
easily
(F)
 • Wikis
(F)
 • LaTex
(S)
 • Smart
Tutor
(S)
 • Web
conferencing
(S)
 • Resume
function
on
quizzes
(S)
 • Consistent
look/structure
between
courses
(S)
 • Picture
roster
(S)
 
 Accessiblity
 • Faculty
and
students
confused
this
with
navigation
and
stability
 
 Overall
Improvements
 Login
 • too
many
clicks
(F)
(S)
 • should
be
more
like
shared
drives
(accessible
when
log
into
 computer
on
campus)
(F)
 Mobile
 • need
mobile
app
(F)
 
 • push
notifications,
RSS
feeds
(F)
(S)
 Public
 • need
ability
to
make
parts
public
(resources,
etc)
(F)
(S)
 Migration
 • easy
porting
to
new
system
(F)
 Listings
 • make
default
-
latest
entry
on
top
(for
anything)
(F)
 MyCourses
 • elearning
could
be
more
integrated
with
other
UTD
systems
 
 • ability
to
open
multiple
windows
 
 • media
library
that
allows
students
to
post/upload
(S)
 
 • site
map
(S)

• 5.
  Results
from
Back-end
Users
Focus
Group
 
 Delivered
integration
w/
PS
 • functionality
delivered
-
50%
(MH)
 • API
hooks
(SH)
 • Tight
integration
(NTH)
 Structure
/
hierarchy
 • Detail
level
-
product
/
service
-
how
much
control
 do
we
have
(MH)
 • Conditional
course
existence
(MH)
 • Roles
-
admin,
support
-
different
levels
(MH)
 Authentication
 • LDAP
(MH)

-
Shibboleth
(SH)
 • Point
of
entry
control
-
single
sign
on
(NTH)
 Hardware
 • No
specific
hardware
as
long
as
below
met
 • Enterprise
level
(MH)
 Database
 • No
flat
architecture
-
Oracle/
MS
SQL
(MH)
 • Must
be
able
to
query
(MH)
 • Published
schemes
(SH)
 External
integration
control
 • We
know
where
our
data
resides
(MH)
 • We
know
who
has
access
to
our
data
(MH)
 Archiving
 • Comprehensive
backup
strategy
(MH)
 • Capability
for
restoring
deleted
content
(MH)
 • Daily
backup
(SH)
 Application
 • Web-based
(MH)
 • Support
multiple
browsers/OS
(MH)

 • High
availability
(MH)
 Migration
 • Migration
support
from
Vista
(SH)
 Security
 • SSL
-
for
mobile
(grades)
(MH)
 Hosting
 • Meet
UTD/state
requirements
(MH)
 Integration
Control
of
content
 • Course
object
(MH)
 • Individual
object
(MH)
 • Gradebook
(MH)
 • Member
enrollment
(MH)
 • Term-based
(MH)
 Support
 • What
kind
(MH)
 • Multiple
levels
(MH)
 • Process
(MH)
 • SLAs
(MH)
 Hosting
 • Data
storage
meets
FERPA,
UTD,
state
policies
 (MH)
 Concern-timeline
for
vendors
to
get
through
the
vetting
process
in
Information
Security.

• 6.
  Appendix
A
 
 Student
Focus
Group
Survey
 
 What is yourstatus? Freshman Sophomore Junior Senior Graduate Student What is your major / school? How long have you been using eLearning? How comfortable are you using eLearning? Very comfortable Comfortable Neutral Uncomfortable Very uncomfortable Which course tools in eLearning do your instructors use? • Announcements • Goals • Assessments • Gradebook o Quizzes • Grading forms (rubrics) o Surveys • Groups o Self-tests • Learning Modules • Assignments • Mail o Individual assignments • Posting syllabus o Group assignments • Web links • Calendar • Who’s Online • Chat / Whiteboard • Discussions o Graded discussions o Journals o Blogs o Threaded Which student tools do you use? • My Files • My Grades • My Progress • Notes Please use the back of this survey for any other comments you’d like to add.
• 7.
  Student
Focus
Group
Survey

 
 What
is
your
status?
 [2]Freshman
 [1]
Sophomore
 
 [2]
Junior
 [1]
Senior
 [8]
Graduate
Student
 What
department
are
you
in?

 • [2]
Arts
and
Humanities
 • [1]
Natural
Sciences
and
Mathematics
 • [4]
School
of
Management
 • [1]
Economic
Public
Policy
Sciences
 • [0]
Brain
and
Behavioral
Sciences
 • [4]
Engineering
and
Computer
Sciences
 • [1]
Undeclared
 How
long
have
you
been
using
eLearning?

 • [9]
1
year
 • [1]
3
years
 • [1]
5
years
 How
comfortable
are
you
with
eLearning
(Please
mark
your
response)
 















[3]
Very
Comfortable
 















[6]
Comfortable
 















[5]Neutral
 















[]
Uncomfortable
 















[]
Very
uncomfortable
 

 What
tools
are
you
currently
using
in
eLearning
(Mark
all
that
apply)
 















[10]
Announcements
 
 
 
 
 















[2]
Assessments
 
 
 
 
 [1]
Goals
 































[1]Quizzes
 
 
 
 
 [9]
Grade
Book
 































[]
Surveys
 
 
 
 
 [3]
Grading
Forms
(Rubrics)
 































[1]
Self‐Tests
 
 
 
 
 [5]
Groups
 















[4]Assignments
 
 
 
 
 [6]
Learning
Modules
 































[7]Individual
Assignments
 
 
 [11]
Mail
 































[7]Group
Assignments
 
 
 
 















[1]Calendar
 
 
 
 
 
 [12]
Syllabus
 















[4]Chat/Whiteboard
 
 
 
 
 
 















[4]Discussions
 
 
 
 
 
 































[3]Graded
discussions
 
 
 
 [3]
Web
Links
 































[2]Journals

 
 
 
 [3]
Who’s
Online
 































[4]Blogs
 































[8]Threaded
 

 Which
student
tools
do
you
use?
 
 [6]
My
Files
 
 
 
 
 
 [11]
My
Grades
 
 []
My
Progress
 
 
 
 
 
 [2]
Notes
 

• 8.
  
 
 Please
use
the
back
of
this
survey
for
any
other
comments
you’d
like
to
add.
 
 • I
have
never
had
a
semester
where
every
professor
used
eLearning.

As
of
now,
I
 have
2
using
eLearning,
2
using
their
own
personal
websites,
and
out
one
using
the
 library’s
eReserves.

For
this
type
of
software
to
be
effective,
use
needs
to
be
 mandatory
by
professors.

As
of
right
now,
it
is
simply
an
inconvenience
to
use
the
 software.

Currently,
the
math
and
science
departments
have
started
using
 computerized
testing.

This
software,
much
learning,
while
primarily
a
 testing/quizzing
software,
should
have
the
ability
to
post
 notes/announcements/grades
it
would
be
much
better
for
students
if
there
was
one
 type
of
software
used
throughout
the
university.

I
imagine
it
would
also
save
quite
a
 bit
of
money
on
licensing
fees
(the
math
department
has
already
purchased
6000
 copies
of
this
software.

It’s
already
very
difficult
to
get
a
professor
to
learn
one
new
 piece
of
software
–
having
one
central
software
would
greatly
help
its
adoption.
 
 • eLearning
is
fairly
simple
to
use
and
I
like
it
but
in
all
honesty
eLearning
doesn’t
 look
very
cool.
 
 • Library
integration.

I
am
also
working
at
UT
Southwestern
Medical
center
 (Research)
Academic
Information
Systems.

We
use
UPIC
(PeopleSoft
now
Oracle)
to
 write
eLearning
and
we
have
ELM
(Enterprise
Learning
Management).

I
write
 technical
training.

We
are
looking
for
solutions
as
well.

Suggestions:

Saiki,
Moodle
 Rooms
 
 • I
use
eLearning
as
an
instructor
(TA).

Very
little
use
of
eLearning
as
a
student
 
 • I
am
currently
doing
an
independent
study,
so
I
am
not
in
a
“class”
that
uses
 eLearning.

However,
I
am
a
TA
for
a
professor
who
uses
it
only
a
little
(to
post
 assignments
and
syllabus).

I
am
assigned
to
teach
next
semester
though,
and
I
will
 be
using
it
a
lot
–
the
class
will
be
a
“hybrid”
(half
on‐line,
half
in
person)
class.

• 9.
  
 Faculty
Focus
Group
Survey

 
 What
is
your
role
in
the
university?

 [28]
Faculty
 [2]
Staff
 
 
 What
department
are
you
in?

 [3]
Arts
and
Humanities
 [3]
Brain
and
Behavioral
Sciences
 [3]
Natural
Sciences
and
Mathematics
 [3]
Engineering
and
Computer
Sciences
 [10]
School
of
Management
 [1]
Interdisciplinary
Studies
 [5]
Economic
Public
Policy
Sciences
 
 
 How
long
have
you
been
using
eLearning?

 [2]
Never
 [5]
5
years
 [7]
1
year
 [1]
8
years
 [8]
2
years
 [1]
9
years
 [2]
4
years
 
 
 How
comfortable
are
you
with
eLearning
(Please
mark
your
response)
 [12]
Very
Comfortable
 [1]Neutral
 []
Uncomfortable
 [14]
Comfortable
 [2]Very
uncomfortable
 
 
 What
tools
are
you
currently
using
in
eLearning
(Mark
all
that
apply)
 [22]
Announcements
 [13]
Calendar
 [13]
File
manager

 [22]
Mail
 [8]
Assessments
 [7]
 [3]
Goals
 [22]
Syllabus
 [13]
Quizzes
 Chat/Whiteboard
 [26]
Grade
Book
 [1]
SCORM
 [5]
Surveys
 [17]
Discussions
 [4]
Grading
Forms
 [9]
Templates
 [8]
Self‐Tests
 [9]
Graded
 (Rubrics)
 [17]
Web
Links
 [11]
Assignments
 discussions
 [11]
Groups
 [11]
Who’s
Online
 [15]
Individual
 [4]
Journals
 
 [14]
Learning
 
 Assignments
 [5]
Blogs
 Modules
 [11]
Group
 [21]
Threaded
 [13]
Selective
 Assignments
 
 Release
 
 Please
add
any
other
comments
below.
 • I’m
interested
in
open
source
products
like
Sakai
 • It
is
a
great
resource.

My
use
depends
on
the
course
I
am
teaching
and
the
level
of
the
 course
(under
graduate
or
graduate)
 • I
have
helped
in
past
webCT
/
eLearning
initiatives
and
appreciate
this
chance
for
us
to
 share
our
experiences.

Please
don’t
hesitate
to
ask
if
I
can
help
in
anyway.

I
feel
like
the
 current
tool
we
are
using
for
courses
is
difficult
for
students
and
instructors
to
use

easily
 and
I
am
excited
to
explore
other
opportunities
for
us
to
improve
our
distance
learning
and
 enhance
learning
in
all
UTD
courses.
 • 1.

When
grading
assignments
submitted
in
eLearning,
it
should
be
possible
to
go
straight
 from
one
paper
to
the
next
without
having
to
go
back
to
the
main
screen
for
that

• 10.
  assignment.

2.

When
Iooking
at
a
list
of
submitted
assignments
and
sort
it
by
student
 name,
I
find
two
problems
a.

The
list
is
sorted
by
first
name,
not
last.

B.

When
I
look
at
a
 paper
and
then
go
back
to
the
list,
the
sorting
reverts
back
to
default.

Resorting
takes
 forever.

I
have
120
students
per
class
submitting
at
least
10
papers
each,
in
3
courses.

3.

 Students
simply
don’t
know
how
to
use
eLearning
features.

They
don’t
know
how
to
edit
a
 submitted
assignment,
for
example.

They
don’t
understand
the
difference
between
a
public
 discussion
and
email.

A
more
accessible
system
might
cause
less
confusion
in
students
with
 more
limited
computer
skills.

4.

Links
to
TurnItIn
and
CPR
would
be
helpful.

Integration,
I
 mean.

5.

If
a
TA
posts
an
announcement,
I
have
no
access
to
edit
or
delete
it.

I
should
be
 able
to
change
all
announcements.

Similarly,
students
can
change
the
calendar
for
 everyone,
and
I
can’t
intervene.

I
had
a
student
post
some
of
her
assignments
there,
and
I
 couldn’t
delete
them.

6.

In
assignments,
HTML
Enabler
does
not
work
for
most
students,
 which
causes
widespread
panic.

7.
The
eLearning
help
number
is
not
helpful.

On
campus
 support
couldn’t
be
better,
but
I
hear
many
bitter
students
complain
about
the
remote
help
 desk.
 • I
have
anywhere
from
12
to
17
sections
of
one
course.

The
most
glaring
omissions
in
 eLearning
is
the
lack
of
centralized
location
where
I
can
store
files,
quizzes,
assessments,
 etc.

that
pertain
to
all
the
sections.

I
would
also
like
to
send
mail
and
post
announcements
 from
one
place
where
I
can
send
it
to
all
the
sections
simultaneously.

Likewise,
I
would
like
 to
post
assessments
in
one
place
where
they
can
be
accessed
by
all
sections.

With
all
of
the
 above,
I
still
would
like
to
retain
the
individual
sections
because
when
it
comes
to
posting
 grades
in
Orion,
I
can
just
draw
them
from
the
individual
sections
in
eLearning.

Bunching
 all
the
sections
into
a
mega
section
would
not
work
for
me
because
for
grades
I
still
need
to
 retain
individual
sections.

Also,
different
sections
are
taught
by
different
TAs,
who
need
to
 have
access
only
to
their
sections.
 • Keep
it
simple!

A
greater
number
of
instructors
and
students
will
use
it.

Basic
Blackboard
 worked
very
well
at
the
University
of
Arizona
–
Eller
College
(Walter
is
the
contact
if
you
 would
like
to
call
him)
 • I
have
not
found
the
current
eLearning
system
to
be
useful
so
I
do
not
use
it.

I
would
hope
 we
could
find
a
more
effective
program
for
the
university.
 • I
would
love
to
see
a
mobile
application
for
the
system
–
particularly
discussions
and
 quizzes.
 • Keep
hoping
for
portfolio
developments

• 11.
  
 Appendix
B
 
 Focus Group GuidingQuestions Let’s talk about your Current usage • What is your level of expertise with eLearning? • What do you use eLearning for? • What do you like about eLearning? Let’s talk about Usability • What do you think about eLearning’s interface? Explain. • What improvements do you need? Let’s talk about Tools in a Learning Management System • What do you like about the tools you are currently using in eLearning? • What do you not like about these tools? • Which tools are essential? (need to be part of whatever system we use?) Let’s talk about Communication • How do you communicate electronically with your students? • How do you use eLearning to communicate with your students? • What is missing from eLearning that would enable you to more effectively communicate with your students? Let’s talk about Collaboration • How do your students collaborate electronically? • How do you use eLearning for collaboration? • What is missing from eLearning that would enable you and your students to more effectively collaborate? Let’s talk about Assessment • How do you assess your students electronically? • How do you use eLearning to assess your students? • Are you able to assess your students effectively using eLearning? • How could you better assess your students in an online format? Let’s talk about Learning Modules • How do you organize your course (in eLearning)? • What content tools in eLearning do you use? • How would you rate each of these content tools and why? Let’s talk about Course Management • What course management tools in eLearning do you use? • How would you rate each of these course management tools and why? • What course management tools would you like to see made available?
• 12.
  Let’s talk aboutGradebook • What do you need in an LMS gradebook? • What do you like about the eLearning gradebook? • What do you not like about the eLearning gradebook? • What features would you like to see added (or changes made to) the gradebook? Let’s talk about Integrated Tools • What integrated tools do you use (i.e. Respondus, Turnitin, eInstruction-clickers)? • What external tools would you like to see integrated with an LMS? What additional tools or what functionality would you like to see in an LMS? (eg portfolios, wikis, etc) Let’s talk about Accessiblity • Do you feel eLearning is accessible? 
 • Why/why not?
 Let’s talk about Overall Improvements • How can eLearning (an LMS) be better?